当前位置:网站首页>Two ways of redis persistence -- detailed explanation of RDB and AOF
Two ways of redis persistence -- detailed explanation of RDB and AOF
2022-06-25 11:40:00 【Big head ice cream】
REDIS Two ways of persistence
RDB
Save the data generation snapshot on disk
command
bgsave:fork Action create subprocess , Short blocking time 
Trigger mode :
- redis The configuration file /etc/redis.conf Set automatic persistence in

save 900 1 each 900 Every second is modified automatically bgsave
3. When performing a full copy from a node , The master node automatically bgsave Generate RDB The file is sent to the slave node
4. perform debug reload Can also be triggered when save operation
5. By default , perform shutdown when , If it's not on AOF Persistence is performed automatically bgsave Into the RDB Persistence
bgsave Process description :
- The parent process determines whether a child process is persisting , Exit if it exists
- fork operation , After the child process is generated, it returns :background saving started. The parent process has been released .latest_fork_usec It's the last time fork Operation time , Company ( subtle )
- The child process starts to create RDB file :lastsave You can see the last time save Time for
- When finished, the child process sends a signal to the parent process , Parent process statistics update info persistence
save: Blocking redis, Has been abandoned 
RDB file
- Storage location :config get dir; config get dbfilename
- Compressed storage is recommended to be enabled :config set rdbcompression yes
- Check whether the file is damaged : Tools redis-check-dump
RDB Advantages and disadvantages
Compact binary , Load fast , Applicable to disaster recovery . But using subprocesses is a heavyweight operation and is not suitable for real-time persistence , And the old version is not compatible . So there is. AOF(append only file) Persistence mode
AOF
Record each write command in an independent log , Re executing when restarting . It's solved redis The real-time problem of data persistence .
Turn on use
Turn on :config set appendonly yes
file name :config get dir ;config get appendfilename; The default is appendonly.aof
technological process :
- All write commands are appended to aof_buf Buffer zone
- AOF The buffer makes synchronous operation to the hard disk according to the corresponding policy
- Need to be right aof Files periodically overwrite compressed space
- redis Load for data recovery when the server restarts
Command write AOF File format
Text protocol format
for example :set hello world
write in AOF In the file :*3\r\n$3\r\nset\r\n$5\r\n\hello\r\n$5\r\nworld\r\n
advantage : Can be read , Easy to modify ; Avoid secondary treatment ; Compatibility is good.
File synchronization policy
By the parameter appendfsync control
- always: Command write buf After calling the system fsync The operation is synchronized to AOF file , After completion, the thread returns .( Don't suggest )
- everysec: write in aof_buf After the call write The operation returns... After writing to the buffer . Call every second fsync operation .( Default , At most 2s The data of )
- no: write in aof_buf After the call write The operation returns... After writing to the buffer . Synchronization to the hard disk is done by the operating system , Usually 30s.( Security cannot be guaranteed , Don't suggest )
fsync operation : Force synchronization to hard disk , After writing to the hard disk, the thread returns , Ensure persistence .
write operation : When the file buffer page is written, it returns , When the buffer is full or reaches certain conditions, it is persisted . If an intermediate outage occurs , Data in buffer pages may be lost .
Rewrite mechanism
Convert the data in the process into a write command and write it again AOF file , Compress AOF file size , It can load faster .
Trigger mode :
Manual trigger :bgrewriteaof
Automatic triggering : According to the parameters :auto-aof-rewrite-min-size( function AOF When rewriting aof Minimum file size ) and auto-aof-rewrite-percentage( At present aof The file size aof_current_size And the volume of the last file rewritten aof_base_size The ratio of the ) The parameter determines the trigger time
When aof-current-size>auto-aof-rewrite-min-size
And (aof_current_size - aof_base_size)/aof_base_size>=auto-aof-rewrite-percentage
ps: aof_current_size and aof_base_size adopt info persistence see
Rewrite process :
- The parent process checks to see if the fork operation , If not, create a child process
- The parent process returns in response to other commands , All operation records enter AOF buffer , And according to appendfsync Policy synced to hard disk ; The new data of the child process is written to AOF Rewrite buffer
- The subprocess writes the command merge rule to the new... According to the memory snapshot AOF file , Each batch write to the hard disk , According to the parameters aof-rewrite-incremental-fsync control , Default 32M.
- new AOF When the file is written , Parent process update info Statistics
- The parent process put AOF The data in the file rewrite buffer is written to the new AOF In file
- Replace old and new files , Finish rewriting .
Restart the loading mechanism
Priority load AOF file , If not, let's see RDB file
File check :
If the wrong file cannot be loaded, it can be backed up and repaired :redis-check-aof --fix
Power failure and other faults may cause aof Incomplete writing at the end of the file . Parameters aof-load-truncated To be compatible with this situation , Default on , If you encounter this problem, ignore and continue loading , Print warning log .
Problem optimization and positioning
fork operation
fork The operation will copy the spatial memory page table of the parent process . for example 10G Of redis The process needs to replicate about 20M Memory page table for . so fork The operation time is closely related to the total memory of the process .
fork Time consuming problem location
Under normal circumstances fork Operation per GB Time consuming 20ms about .
Check out the last fork Operating time :info stats in latest_fork_usec( Company : Microsecond )
How to improve fork Operating time :
- Using a physical machine , Avoid using Xen
- control redis Instance maximum available memory , Suggest 10G within .
- Reduce fork frequency , Moderate relaxation AOF Automatic trigger time , Avoid unnecessary full replication .
Subprocess overhead
cpu
- redis yes cpu Intensive services , Child processes can consume a single core cpu Of 90%, So don't do cpu Single core binding .
- Ensure that only one child process performs the rewrite
Memory
Child process passed fork Operation produces , The amount of memory used is equal to that of the parent process , In theory, we need 2 Times the memory to complete the persistence operation . however linux There is a copy on write mechanism (copy-on-write),
The parent and child processes share the same physical memory pages , When the parent process processes the write request, it will create a copy of the modified memory page , The subprocess is in fork Share the whole memory snapshot of the parent process in the process .
Rewrite memory consumption :
monitor redis journal
RDB rewrite :copy-on-write Copy memory created by the parent process
AOF rewrite :copy-on-write Copy memory created by the parent process +AOF Memory occupied by buffer
Memory consumption optimization :
- Try to ensure that only one child process is working
- Avoid doing subprocess rewriting when writing a lot , This will cause the parent process to create too many copies of memory pages , Memory consumption is large
- linux Yes TransparentHugePage( Default on ), Big page 2M, When opened, the duplicate page will be copied from 4k Change to 2M, It will greatly increase the memory consumption during rewriting .
Close the way :sudo echo never > /sys/kernel/mm/transaparent_hugepage/enabled
Hard disk
Persist to hard disk , Hard disk write pressure , command :sar/iostat/iotop Analyze the current hard disk load .
How to optimize :
- Don't deploy with other services with high hard disk load , Like storage services 、 Message queue
- Open configuration :no-appendfsync-on-rewrite, Indicates that it is not enabled during rewriting fsycnc operation
- When open AOF Functional redis And when writing scenes with high traffic , The throughput of ordinary mechanical disks is 100MB/s. The bottleneck is AOF Sync on hard disk
- If it is more redis example , Can be AOF Files are stored on different disks , Apportion write pressure
AOF Add blocking
AOF The common hard disk synchronization strategy for persistence is everysec, If the system hard disk resource is busy , Can cause redis Main thread blocking .
Main thread every 1 Second for a comparison , If the time since the last successful synchronization >2s, Blocking the main thread
therefore
- everysec May lose 2s The data of
- Busy hard disk resources may lead to redis The main thread is blocked
Problem location :
- happen AOF Blocking time ,redis Relevant contents will be output in the log
- When blocking occurs ,info persistence The statistics aof_delayed_fsync The indicators will add up
- iotop You can see the high load problem of the hard disk
Reference resources :《redis Development and operations 》 Fu Lei 、 Written by zhangyijun
边栏推荐
- Source code analysis of AQS & reentrantlock
- Compilation of learning from Wang Shuang (1)
- Comparator (for arrays.sort)
- 杭州/北京内推 | 阿里达摩院招聘视觉生成方向学术实习生(人才计划)
- CFCA安心签接入
- SystemVerilog (XIII) - enumerate data types
- 某APP中模拟器检测分析
- Handler、Message、Looper、MessageQueue
- Hangzhou / Beijing neitui Ali Dharma academy recruits academic interns in visual generation (talent plan)
- ARM64特有一些的汇编指令
猜你喜欢

基于Minifilter框架的双缓冲透明加解密驱动 课程论文+项目源码

時創能源沖刺科創板:擬募資11億 年營收7億淨利反降36%

ThingsPanel 发布物联网手机客户端(多图)

Comment TCP gère - t - il les exceptions lors de trois poignées de main et de quatre vagues?

查询法,中断法实现USART通信

牛客网:主持人调度

Shichuang Energy sprint Technology Innovation Board: le chiffre d'affaires annuel prévu de 1,1 milliard de RMB est de 0,7 milliard de RMB, en baisse de 36%

Compilation of learning from Wang Shuang (1)

Spark runs wordcount (case 2)

按钮多次点击造成结果
随机推荐
GCC related
try-catch-finally
Keywords serializable serialization and deserialization
杭州/北京内推 | 阿里达摩院招聘视觉生成方向学术实习生(人才计划)
Use of Presto visualization client-yanagishima20.0
GC
金太阳教育美股上市:市值3.6亿美元 成小盘中概股
CFCA安心签接入
Research on parallel computing architecture of meteorological early warning based on supercomputing platform
Big Endian 和 Little Endian
Comparable的使用(用于Arrays.sort)
Builder pattern
What are the ways to simulate and burn programs? (including common tools and usage)
建造者模式
Comparator(用于Arrays.sort)
CMU puts forward a new NLP paradigm - reconstructing pre training, and achieving 134 high scores in college entrance examination English
Spark runs wordcount (case 1)
Some assembly instructions specific to arm64
Niuke: rotation array
Arrays.asList()