当前位置:网站首页>How to use rdbtools to analyze redis large keys
How to use rdbtools to analyze redis large keys
2022-06-24 17:26:00 【Yu Hang】
Big key Hazards of
be familiar with redis Our friends all know , Big key It is a risk that should be avoided in the design and practice of the application . Big key There are many dangers , for example :
- Causing traffic to reach a bottleneck
- Cause the tilt of the partition capacity
- What is more serious is reading and writing key Cause the system to be unresponsive , Generate a large number of slow queries and so on
Generally we think that it is more than 10kb The size of string Type of key Or more than 1 Ten thousand elements set It can be called big key, A reasonable split should be made
Big key Acquisition
How to get large key For those who use cloud database , Is a relatively easy thing , Because many cloud manufacturers provide large by default key Analysis tools , For example, Tencent cloud database is in 【 Console 】【 System monitoring 】【 Monitoring Overview 】 The page provides a large key Analysis function , The principle is to analyze the static state RDB The file is then pulled out of it key Sort by size , The focus of this article is to share how to use open source tools rdbtools Carry on the big key analysis
rdbtools Installation
rdbtools There are three main functions
- Analyze static rdb File and generate csv Format memory report
- take rdb File dump becomes json Format
- utilize diff The tool compares two rdb The difference of documents Now let's start
1. The first step is to install python and pip
Because the author's operating system uses centos8.0, By default python3 and pip3 So there is no need for additional installation , Readers can install by themselves , There are many online tutorials , I won't repeat .
>>> import sys >>> sys.version '3.6.8 (default, May 21 2019, 23:51:36) \n[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]' >>>
2. obtain rdb file
build by oneself redis Students can use redis-port Toolset export rdb file . Friends who use cloud database can contact the cloud manufacturer to get the download link .
Export directly to local or use rz/sz Import into your own machine
[[email protected] ~]# yum install lrzsz Last metadata expiration check: 0:10:21 ago on Thu 18 Mar 2021 04:31:25 PM CST. Dependencies resolved. ======================================================================================================================================================================================== Package Arch Version Repository Size ======================================================================================================================================================================================== Installing: lrzsz x86_64 0.12.20-43.el8 BaseOS 84 k Transaction Summary ======================================================================================================================================================================================== Install 1 Package Total download size: 84 k Installed size: 190 k Is this ok [y/N]: y Downloading Packages: lrzsz-0.12.20-43.el8.x86_64.rpm 2.2 MB/s | 84 kB 00:00 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 2.1 MB/s | 84 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : lrzsz-0.12.20-43.el8.x86_64 1/1 Running scriptlet: lrzsz-0.12.20-43.el8.x86_64 1/1 Verifying : lrzsz-0.12.20-43.el8.x86_64 1/1 Installed: lrzsz-0.12.20-43.el8.x86_64 Complete!
3. install rdbtools
[[email protected] ~]# pip install rdbtools
After installation, pass help Simply understand the command , Be careful --command or -c Is a required parameter
[[email protected] ~]# rdb --help usage: usage: rdb [options] /path/to/dump.rdb Example : rdb --command json -k "user.*" /var/redis/6379/dump.rdb // From dump.rdb Analyze all the data in the file `user.` At the beginning key And the output is jason Format , Print it to the screen positional arguments: // Positional arguments dump_file RDB Dump file to process optional arguments: -h, --help show this help message and exit// View help information for commands -c CMD, --command CMD// Required parameters ,-c json Indicates stored as json Format , The most common is -c memory To generate csv Format memory snapshot , also diff Pattern comparison Command to execute. Valid commands are json, diff, justkeys, justkeyvals, memory and protocol -f FILE, --file FILE // output to a file Output file -n DBS, --db DBS // Specifies that only a number of db, for example db0, If you do not specify that all... Are included by default Database Number. Multiple databases can be provided. If not specified, all databases will be included. -k KEYS, --key KEYS // Specifies the... That analyzes and exports a prefix key, regular expression Keys to export. This can be a regular expression -o NOT_KEYS, --not-key NOT_KEYS // Specifies that a prefix is not parsed and exported key, regular expression Keys Not to export. This can be a regular expression -t TYPES, --type TYPES // Specify which data types to analyze , for example string、hash、set、list、sortedset etc. , Multiple types can be specified , If the default analysis is not specified, all Data types to include. Possible values are string, hash, set, sortedset, list. Multiple typees can be provided. If not specified, all data types will be returned -b BYTES, --bytes BYTES // Specify how much the output exceeds Byte Of key, for example --bytes 10240 Indicates that only output exceeds 10k The size of key Limit memory output to keys greater to or equal to this value (in bytes) -l LARGEST, --largest LARGEST // Output TOP How many key, for example --largest 100, Means output only top 100 Of key Limit memory output to only the top N keys (by size) -e {raw,print,utf8,base64}, --escape {raw,print,utf8,base64} // Specifies the default encoding for the output , Default RAW Escape strings to encoding: raw (default), print, utf8, or base64. -x, --no-expire With protocol command, remove expiry from all keys // Only command by protocol Patterns work , It means not to output all expired key, Only those that never expire key -a N, --amend-expire N // Only command by protocol Patterns work , Indicates to set the expiration time key increase N Seconds past due With protocol command, add N seconds to key expiry time
4. Next, let's briefly analyze
[[email protected] ~]# rdb --command json dump.rdb > ./test WARNING: python-lzf package NOT detected. Parsing dump file will be very slow unless you install it. To install, run the following command: pip install python-lzf
Here we are reminded that there is no python-lzf Packets cause analysis to be slow , Let's install it . You can update it first yum Source , Prevent missing packages
[[email protected] ~]# yum update
Then install again
[[email protected] ~]# pip install python-lzf WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead. Collecting python-lzf Downloading http://mirrors.tencentyun.com/pypi/packages/e3/33/b8f67bbe695ccc39f868ae55378993a7bde1357a0567803a80467c8ce1a4/python-lzf-0.2.4.tar.gz Installing collected packages: python-lzf Running setup.py install for python-lzf ... error Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-go178sdk/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6oauj90r-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_ext building 'lzf' extension creating build creating build/temp.linux-x86_64-3.6 gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I. -I/usr/include/python3.6m -c lzf_module.c -o build/temp.linux-x86_64-3.6/lzf_module.o -Wall gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory error: command 'gcc' failed with exit status 1 ---------------------------------------- Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-go178sdk/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6oauj90r-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-go178sdk/python-lzf/
There is an error gcc An error was reported when compiling , On github Search for , Is not installed
redhat-rpm-config, Then let's install
$ sudo dnf install redhat-rpm-config
Install well
redhat-rpm-configafter , Install again python-lzf Wrong again. , This time, there are few hints Python.h, Check it online , It turned out that this time there was a lack ofpython-devel.
[[email protected] ~]# pip install python-lzf WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead. Collecting python-lzf Downloading http://mirrors.tencentyun.com/pypi/packages/e3/33/b8f67bbe695ccc39f868ae55378993a7bde1357a0567803a80467c8ce1a4/python-lzf-0.2.4.tar.gz Installing collected packages: python-lzf Running setup.py install for python-lzf ... error Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-dxcuwlzv/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9n6nf8r_-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_ext building 'lzf' extension creating build creating build/temp.linux-x86_64-3.6 gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I. -I/usr/include/python3.6m -c lzf_module.c -o build/temp.linux-x86_64-3.6/lzf_module.o -Wall lzf_module.c:3:10: fatal error: Python.h: No such file or directory #include "Python.h" ^~~~~~~~~~ compilation terminated. error: command 'gcc' failed with exit status 1 ---------------------------------------- Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-dxcuwlzv/python-lzf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9n6nf8r_-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-dxcuwlzv/python-lzf/
We directly yum The installation could not find , For example, as shown below , Let's try to search
yum searchonce , eureka ,python36-devel.x86_64, Then install again .
[[email protected] ~]# yum install python-devel Last metadata expiration check: 0:00:58 ago on Thu 18 Mar 2021 05:14:53 PM CST. No match for argument: python-devel Error: Unable to find a match: python-devel
[[email protected] ~]# yum search python3 | grep devel Last metadata expiration check: 0:01:29 ago on Thu 18 Mar 2021 05:14:53 PM CST. python3-qscintilla-qt5-devel.noarch : Development files for QScintilla-qt5 python3 bindings boost169-mpich-python3-devel.x86_64 : Shared library symbolic links for Boost.MPI Python 3 component boost169-openmpi-python3-devel.x86_64 : Shared library symbolic links for Boost.MPI Python 3 component boost169-python3-devel.x86_64 : Shared object symbolic links for Boost.Python 3 python3-TurboGears2.noarch : Next generation front-to-back web development megaframework python3-cherrypy.noarch : Pythonic, object-oriented web development framework python3-hupper.noarch : Integrated process monitor for developing servers python3-idle.i686 : A basic graphical development environment for Python python3-idle.x86_64 : A basic graphical development environment for Python python3-kobo.noarch : Python modules for tools development python3-pycxx-devel.noarch : PyCXX header and source files python3-werkzeug.noarch : The Swiss Army knife of Python web development python36-devel.x86_64 : Libraries and header files needed for Python development python38-devel.x86_64 : Libraries and header files needed for Python development python38-idle.x86_64 : A basic graphical development environment for Python
We have solved these dependencies , We will install it again smoothly
[[email protected] ~]# pip install python-lzf WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead. Collecting python-lzf Downloading http://mirrors.tencentyun.com/pypi/packages/e3/33/b8f67bbe695ccc39f868ae55378993a7bde1357a0567803a80467c8ce1a4/python-lzf-0.2.4.tar.gz Installing collected packages: python-lzf Running setup.py install for python-lzf ... done Successfully installed python-lzf-0.2.4
5. Formal analysis
Here is a common command , analysis rdb The file of top100 The big key, have access to lrzsz Download to local , Use Excel Analyze .
[email protected] ~]# rdb -c memory dump-6120957-redis-server-ignore-140016731-ignore-1-ignore.rdb -l 100 -f ./dump_memory.csv
CSV Field description in
database | type | key | size_in_bytes | encoding | num_elements | len_largest_element | expiry |
|---|---|---|---|---|---|---|---|
key Belongs to db Number | key For example string list set etc. | key name | key size ( Number of bytes ) | Encoding mode (hashtable,ziplist,string etc. ) | key The number of elements in | key Length of the largest element in | Expiration time |
You can also use
LOAD DATA INFILEThe statement is imported into the database , Use SQL Statement analysis , Such as total memory usage of query 、 Query total key Number 、 Query specific type Of key Number and so on . I won't repeat
边栏推荐
- Devops in digital transformation digital risk
- How to learn go language happily? Let's go!
- 让UPS“印象派用户”重新认识可靠性
- Tencent cloud layer 7 load balancing log analysis and monitoring
- zblog系统如何根据用户ID获取用户相关信息的教程
- Tencent cloud database mysql:sql flow restriction
- What is the problem that the data is not displayed on the login web page after the configuration of the RTSP video security intelligent monitoring system easynvr is completed
- Using easyjson to improve the efficiency of serialization transmission
- 跟着Vam一起学习Typescript(第一期)
- TRCT test cloud + article online speed
猜你喜欢
随机推荐
C4D learning notes
On N handshakes and M waves of TCP
"Competition" and "opportunity" hidden in security operation in the cloud Era
Explanation of MySQL indexing principle
03. Tencent cloud IOT device side learning -- overview of mqtt control package
Solutions for RTSP video streaming played by several browsers
NFT元宇宙源码搭建解析与介绍
zblog判断某个插件是否安装启用的内置函数代码
Elastic searchable snapshot function (frozen Tier 3)
区块哈希游戏竞猜系统开发(成熟代码)
FPGA systematic learning notes serialization_ Day8 [design of 4-bit multiplier and 4-bit divider]
Classic examples of C language 100
Research on clock synchronization performance monitoring system based on 1588v2 Technology
Robot toolbox matlab robotics toolbox
"Gambler" bubble Matt turns around
Ramda 鲜为人知的一面
A comprehensive understanding of fiber to home FTTH and optical splitter
Solution to the problem that kibana's map cannot render longitude and latitude coordinate data
Leveldb source code analysis -- version management
About with admin option and with grant option

