An online frequent fullgc troubleshooting

background

One day , A node of a service on the line suddenly gives an alarm , Frequently FullGC, This service hasn't been released or restarted for half a month .

Processing steps

• First isolate the container of the alarm , Remove this node from the gateway , Let the traffic no longer enter this node .
• adopt jmap Snapshot the memory of the node dump Come down .
• adopt jprofiler analysis hprof file .

The analysis reason

There are many tools to analyze hprof file , Such as MAT、JProfiler.
We go through JProfiler analysis , stay JProfiler Open a single snapshot in the startup center of .

Choose to download to local hprof file , Wait for a moment to parse , Then go straight to “ Maximum object ”：

Find a static Field occupied 1851MB size , almost 2GB！！

our JVM The startup parameters have the following settings about memory ：

-Xms8192m 
-Xmx8192m 
-Xmn5120m 
-XX:MetaspaceSize=384m 
-XX:MaxMetaspaceSize=512m 

namely ,java Heap memory allocation 8GB, New generation heap memory 5GB, Then the only thing left for the old age is 3GB, And then that one ConcurrentHashMap Type of static Field , Nearly 2GB（ This big object is obviously placed in the old age ）.
We know JVM When it comes to recycling , It is through reachability analysis to judge whether an object survives , Anything that can be learned from GCRoots Objects that arrive are not recycled , and static The field is GCRoots A kind of , That is to say, our static Fields cannot be recycled . So when it happens FullGC It's always impossible to make room for the elderly , such JVM There will soon be another FullGC, But every time FullGC Can't recycle enough space , In a vicious circle .

solve the problem

By looking at the one in the code static ConcurrentHashMap, Discovery is a field in a class of middleware , Gray scale identification for storing traffic , Only put the content inside , But no deleted code . So if map Of key It's basically different , It will lead to map As time goes by, the content in the becomes larger and larger , Eventually lead to FullGC When , Can't reclaim the memory of the older generation .
This problem , Some time ago, other services have also encountered , Later, it was fed back to the middleware , The middleware side gave a version that solved the problem and let me upgrade .
So why didn't there be any problems when I didn't upgrade before ？ I guess the upstream gateway may have changed the rule of traffic gray identification , Lead to map What's in it key Suddenly it becomes more colorful , such map Can't drop the weight well , Lead to map Growing . At that time, the people who wrote the middleware code probably thought that there would not be so many key, Think map It doesn't get too big , So I didn't delete map The content in .
Checked the version of middleware upgrade , Adopted guava Of cache, Set a maxsize, such map The number of elements in will not exceed maxsize, It will not cause the elderly generation to be occupied so much . This is actually a cache elimination strategy , Keep only the most recently used maxsize Bar record .

reflection

1. JVM Memory problems for , Most of the time, it is not easy to set the heap memory a little larger , If there is a problem with the code , Always run out of memory , At this time, we have to modify the code to solve the problem .
2. Some memory leaks , Not soon , But it happened a long time later , For example, our service , I haven't restarted in half a month , As a result, the elderly generation is slowly occupied and can not be recycled . It seems that increasing the frequency of release may also cover up some hidden memory leakage problems .
3. Local caching can improve speed , But be careful , See if you need to remove unused content , For example, using guava Of cache, Setting a fixed size can effectively avoid memory leakage .