Record and analysis of Rac abnormal heartbeat (ipreamsfails)

environmental information

Oracle 19.12
Oracle Linux 7.9

One ： Real production environment phenomenon

When slightly affected ：

1. The user reported that the database connection failed 、 It lasted for a few minutes and then returned to normal
2. crs The log contains node communication exception information
3. OSWatcher ipReamsfails growth

Severe impact ：

1. The user reported that the database connection failed 、 Production business downtime
2. crs Node kick out , Operating system restart
3. Switch to dg library , Still happen crs Node kick out , Operating system restart
4. Stop database 2 node . The single instance database is running normally , Observe for two weeks without any error .
5. OSWatcher ipReamsfails A lot of growth

Two ： Before testing , Actions taken by the production environment ：

1. Modify the operating system kernel parameters from 4M、3M Add to ,15M、16M（ Official documents ID：2008933.1） The problem remains.
2. Modify the private network connection method , Direct connection with network cable , The problem remains.

3、 ... and ： Start the stress test （ use swingbench） Result

• Test scenario 1 ：

Configuration situation ：
Concurrency number ：200
test result ：
CRS The log contains node communication exception information .
OSWatcher There are findings in the log ipReamsfails Growing .

• Test scenario II ：

Configuration situation ：
Concurrency number ：600
test result ：
CRS The log contains node communication exception information .
OSWatcher There are findings in the log ipReamsfails Growing slowly .

• Test scenario three ：

Configuration situation ：
Adjust the configuration of heartbeat switch , Enable macroframe （Jumbo Frame）
Adjust the heartbeat network card MTU size =9000
Concurrency number ：1000
test result ：
CRS journal normal .
OSWatcher journal ipReamsfails No growth .

• Test scenario 4 ：

Configuration situation ：
Upgrade the network card driver
Adjust the heartbeat network card MTU size =1500
Concurrency number ：600
test result ：
CRS The log contains node communication exception information .
OSWatcher There are findings in the log ipReamsfails Growing slowly .

Four ： Test summary

1. During the test, the pressure is actually produced 10 More than times .
2. No expulsion of database nodes was found .
3. CRS The log contains node communication exception information .
4. OSWatcher There are findings in the log ipReamsfails growth .

5、 ... and ： Final solution

1. Adjust the heartbeat switch configuration to enable macroframe （Jumbo Frame）
2. Adjust the heartbeat network card MTU size =9000

6、 ... and ： Follow up test plan

1. Upgrade the operating system kernel test
2. from Oracle Linux Switch to a Redhat Linux test
3. New server test
4. If conditions permit , The following will supplement the testing of different systems and hardware platforms . If there is no condition, it will be a pigeon .

7、 ... and ： Problem analysis and summary

• IP reassembles failed The phenomenon

1. IP reassembles failed The phenomenon is due to ipfrag_low/high_thresh The cache is full , New packets cannot be received by the operating system , Direct discarding . Produced IpReamsfails Value growth .
2. ipfrag_low/high_thresh buffer Operating system default 3M、4M
   /proc/sys/net/ipv4/ipfrag_low_thresh
   /proc/sys/net/ipv4/ipfrag_high_thresh

• Under what circumstances will network packets be put ipfrag In cache , And cause ipReamsFails Value growth ？

1. In the Internet , Packets will be based on the minimum MTU Size split transmission reorganization . The operating system network card and switch default to 1500 byte .
2. When packets exceed MTU Byte size packets , It will be split into multiple packets and transmitted in turn .
3. After being transmitted to the target end, it will be put ipfrag In cache , Wait for packet reorganization .
4. stay Oracle in , There are two scenes with MTU Size matters .

a) client –> Request and response of database server
Related parameters ：SDU, The default is 8K
b) Heartbeat between database servers and cache fusion Data blocks
Related parameters ：DB_BLOCK_SIZE, The default is 8K

5. By default (MTU 1500 byte ) Data transmission in the service network and heartbeat network will be unpacked and packaged , At this point, the packet will be put ipfrag In cache .
6. Database read operating system network IO, The operating system comes from ipfrag_thresh Read packets from the cache of , Restructuring , Assemble the package into SDU block or DB_BLOCK block .
7. When the operating system starts from ipfrag_thresh The efficiency of reading packets from the cache for packet reorganization is lower than ipfrag_thresh Cache growth efficiency ,ipfrag_thresh The cache will gradually increase , When the maximum cache value is reached , New packet discards , Incomplete packet reception , Package reorganization failed . Report errors ：IP reassembles failed ,IpReamsfails Value growth .

• How to avoid this kind of phenomenon ？

1. To avoid this phenomenon , Through adjustment MTU 9000, Greater than DB_BLOCK_SIZEK 8K size .
2. Transferred between database servers cache fusion Data blocks There will be no unpacking 、 The scenario of packaging , therefore IpReamsfails The phenomenon of growth disappeared .

• Other reasons ？

This phenomenon is related to the system version 、 Network card driver version and other low-level compatibility .

attach ： Official documents and reference descriptions

1. Troubleshooting gc block lost and Poor Network Performance in a RAC Environment (Doc ID 563566.1)
   The direct connection mode of heartbeat network card is adjusted according to the original factory documents 、 Upgrade the network card driver .
2. IPC Send timeout/node eviction etc with high packet reassembles failure (Doc ID 2008933.1)
   Adjusted according to the original factory documents ipfrag_thresh.
3. Recommendation for the Real Application Cluster Interconnect and Jumbo Frames (Doc ID 341788.1)
   Adjusted according to the original factory documents MTU size 9000, The problem did not recur after adjustment .

Those who have encountered this problem can discuss it together , At present, there are still blind spots that have not been covered , For example, the test did not reappear RAC Node expulsion 、 Only adjusted the private network heartbeat , It is not related to APP Adjustment of business flow between .