[dark horse programmer] redis learning notes 002: persistence: RDB and AOF

One 、 Introduction to persistence

What is persistence

Using permanent storage media to save data , The working mechanism of recovering the saved data in a specific event is called persistence

Why persistence

Prevent accidental loss of data , Ensure data security

The persistence process preserves something

• Save the current data state , Snapshot form , Store data results , Storage is simple , The focus is on data -----》 RDB
• Save the operation process of data , Log form , Store operation procedures , The storage format is complex , The focus is on the operation of data ------》AOF
  

Two 、RDB

RDB Starting mode

who , What event , What are you doing ？

• who ：redis Operator （ user ）
• What time? ： instant （ At any time ）
• What are you doing ： Save the data

RDB Mode of starting ——save Instructions

• command

save

• effect
  Manually perform a save operation

RDB Starting mode ——save Instruction related configuration

• dbfilename dump.rdb
  explain ： Set the local database file name , The default value is dump.rdb
  Experience ： Usually set Wei dump- Port number .rdb
• dir
  explain ： Set up storage .rdb Path to file
  Experience ： It is usually set to a directory with large storage space , Directory name data
• rdbcompression yes
  explain ： Set whether to compress data when storing to local database , The default is yes, use LZF Compress
  Experience ： It is usually on by default , If I set it to no, Can save CPU The elapsed time , But it makes the stored files bigger （ huge ）
• rdbchecksumy yes
  explain ： Set whether to RDB File format verification , The verification process is carried out in the process of writing and reading files
  Experience ： It is usually on by default , If set to no, It can save the reading and writing process about 10% Time consuming , However, there is a certain risk of data corruption

Be careful ：Redis It's single threaded , All commands are queued in similar queues , Not recommended save Instructions , because save The execution of the instruction blocks the current Redis The server , Until now RDB Process completion location , It may cause long-term congestion , Online environment is not recommended

Too much data , How to deal with the low efficiency caused by single thread execution ？

The background to perform

• who ：redis Operator （ user ） Initiate instruction ;redis The server controls the execution of instructions
• What time? ： instant （ launch ）; Reasonable time （ perform ）
• What are you doing ： Save the data

RDB Starting mode ——bgsave Instructions

• command

bgsave

• effect
  Start the background save operation manually , But not immediately

RDB Starting mode ——bgsave How commands work

Be careful ：bgsave The order is for save Optimization of blocking problem .Redis It's all about RDB All operations use bgsave The way ,save The command can be discarded

RDB Starting mode ——bgsave Instruction related configuration

The following is followed by save The configuration is the same

newly added

• stop-writes-on-bgsave-error yes
  explain ： If an error occurs in the background stored procedure, the thread , Whether to stop the save operation
  Experience ： It is usually on by default

To present position ,save and bgsave Are manual save instructions , Then the following questions will be introduced

Execute the Save command repeatedly , Forget what to do ？ I don't know how much the data has changed , When to save ？

Automatic execution

• who ：redis The server initiates an instruction （ Based on the conditions ）
• What time? ： Meet the conditions
• What are you doing ： Save the data

RDB Starting mode ——sava To configure

• To configure

save second changes

• effect
  Meet the time limit ke When the number of changes reaches the specified number, persistence is carried out
• Parameters
  second： Monitor the time frame
  changes： monitor key Amount of change
• Location
  stay conf File to configure
• Example
  

RDB Starting mode ——save Configuration principle

Be careful ：
save The configuration should be set according to the actual business situation , If the frequency is too high or too low, performance problems will occur , The results can be catastrophic
save In the configuration second And changes Settings usually have complementary correspondence , Try not to set up an inclusive relationship
save After the configuration is started, the execution is bgsave operation

RDB Comparison of three startup modes

rdb Special starting form

• Copy in full
  It will be mentioned in master-slave replication
• Restart the server while it is running
  debug reload
• Specify to save data when shutting down the server
  shutdown save

RDB Advantages and disadvantages

RDB advantage

• RDB Is a compact binary , High storage efficiency
• RDB What's stored inside is redis A snapshot of data at a point in time , Perfect for data backup , Full scale replication and other scenarios
• RDB It's faster to recover data than AOF Much faster
• application ： Every X Hours to perform bgsave Backup , And will RDB Copy the file to the remote user , For disaster recovery

RDB shortcoming

• RDB Whether it's executing instructions or using configuration , Unable to achieve real-time persistence , It is more likely to lose data
• bgsave The instructions are executed every time they run fork Action create subprocess , To sacrifice some performance
• Redis In many versions of RDB Unified version of file format , There may be incompatibility of data formats between version Services

3、 ... and 、AOF

RDB The shortcomings of

RDB The disadvantages of storage

• A large amount of data is stored , Low efficiency —— Based on the idea of snapshot , Every read and write is all data , When the amount of data is huge , Very inefficient
• Big data IO Low performance
• be based on fork Create child process , Extra consumption of memory
• The risk of data loss caused by downtime

Solutions

• Don't write full data , Only part of the data is recorded
• Change the recorded data without recording the operation process
• Record all operations , Eliminate the risk of data loss
• This is the same. AOF The introduction of

AOF Concept

• AOF Persistence ： Each write is logged as a separate log , Reexecute on reboot AOF The command in the file achieves the purpose of recovering data . And RDB Compared with the process of recording data generation
• AOF Its main function is to solve the real-time problem of data persistence , So far Redis The mainstream way of persistence

RDB The disadvantages of storage

• A large amount of data is stored , Low efficiency —— Based on the idea of snapshot , Every read and write is all data , When the amount of data is huge , Very inefficient
• Big data IO Low performance
• be based on fork Create child process , Extra consumption of memory
• The risk of data loss caused by downtime

Solutions

• Don't write full data , Only part of the data is recorded
• Change the recorded data without recording the operation process
• Record all operations , Eliminate the risk of data loss
• This is the same. AOF The introduction of

AOF Concept

• AOF Persistence ： Each write is logged as a separate log , Reexecute on reboot AOF The command in the file achieves the purpose of recovering data . And RDB Compared with the process of recording data generation
• AOF Its main function is to solve the real-time problem of data persistence , So far Redis The mainstream way of persistence

AOF The process of writing data

AOF Three strategies for writing data

• always( Every time )
  Each write operation is synchronized to AOF In file , Data zero error , Low performance
• everysec( Per second )
  Synchronizes the instructions in the buffer to per second AOF In file , High accuracy of data , Higher performance
  Then the system is suddenly lost 1 Seconds of data
• no( System control )
  Every time the operating system synchronizes to AOF The period of the document , The whole process Out of control

AOF Function of open

• To configure

appendonly yes|no

• effect

  Open or not APF Persistence function , The default is not on

• To configure

appendfsync always|everysec|no

• effect
  AOF Write data strategy

AOF Related configuration

• To configure

appendfilename filename

• effect
  AOF Persistent filename , Default file name appendonly.aof, Recommended configuration is appendonly- Port number .aof
• To configure

dir

• effect
  AOF Persistent file save path , And RDB Just keep persistent files consistent

AOF Problems with writing data

What should be done if the following instructions are executed continuously

AOF rewrite

As the command continues to write AOF, The file will get bigger and bigger , To solve this problem ,Redis introduce AOF Rewrite mechanism to compress file volume ,AOF File rewriting is to Redis In process data is converted to write commands and synchronized to the new AOF Documentation process , Simply put, it is to convert the execution results of several commands of the same data into the instructions corresponding to the final result data for recording

AOF Rewriting effect

• Reduce disk usage , Improve disk utilization
• Improve persistence efficiency , Reduce persistent write time , Improve IO performance
• Reduce data recovery time , Improve data recovery efficiency

AOF Rewriting rule

• Data that has timed out in the process is no longer written to the file
• Ignore invalid instructions , In process data generation is used when rewriting , So new AOF Write command to keep only the final data in the file 　
  Such as del key1,hdel key2,srem key3,set key 222 etc.
• Multiple commands for unified data are merged into one command
  Such as lpush list1 a ,lpush list1 b,lpush list1 c Can be converted to lpush list1 a b c
  In order to prevent client buffer overflow caused by excessive data , Yes list,set,hash,set Other types , Each instruction can write at most 64 Elements

AOF Override method

• Manual rewriting

bgrewriteaof

• Automatic override

auto-aof-rewrite-min-size size
auto-aof-rewrite-percentage percentage

AOF Manual rewriting ——bgrewriteaof How commands work

AOF Auto rewrite mode

• Automatically override trigger condition settings

auto-aof-rewrite-min-size
auto-aof-rewrite-percentage percent

• Automatically rewrite trigger comparison parameters （ Operation instruction info Persistence Get specific information ）

aof_current_size
aof_base_size

• Automatically override trigger conditions
  

AOF Rewrite process

AOF and RDB The difference between

RDB and AOF A sense of choice

• Very sensitive to data , The default... Is recommended AOF Persistence scheme
  AOF Persistence strategy uses erverysecond, Every second fsync once . The strategy redis It can still maintain good processing performance , When something goes wrong , At most 0-1 Data in seconds .
  Be careful ： because AOF The file storage volume is large , And the recovery of data is slow
• Data presentation phase validity , It is recommended to use RDB Persistence scheme
  Data can be well done in the phase without loss （ This stage is manually maintained by developers or operation and maintenance personnel ）, And the recovery speed is faster , Phase point data recovery usually uses RDB programme
  Be careful ： utilize RDB Implementing compact data persistence makes Redis Very low
• Comprehensive comparison

1. RDB And AOF The choice is actually a trade-off , Each has its pros and cons
2. If you can't afford data loss within a few minutes , Very sensitive to business data , choose AOF
3. If you can withstand data loss within a few minutes , And pursue the recovery speed of large data sets , choose RDB
   Disaster recovery options RDB
4. Double insurance strategy , At the same time open RDB and AOF, After restart ,Redis priority of use AOF To recover data , Reduce the amount of lost data

Persistent application scenarios