Record the troubleshooting process of excessive CPU usage

Today, the test feedback online test environment has almost no data in the warehouse , Recently, it happened that the new version was tested , It is estimated that what has affected , Start with the deployment environment troubleshooting .

top Command to view the operation of the server , It is found that a process has been occupying CPU,CPU Consumption basic 100%.

pwdx Check the process , It is found that the process is just the application of the module .

View the thread list of this process , Let's see what thread is taking up CPU Too high ：ps -mp 5866 -o THREAD,tid,time.

You can see that there is a thread CPU The occupation is absurdly high , Logging thread id, The thread id Turn into 16 Base number ,jstack Print the stack of the process , Find the corresponding thread .

From the output, you can see the specific blocked threads , Go to the specific code , It is found that the processing of getting elements from a blocking queue is unreasonable , There's a dead cycle , Lead to CPU Idle .

Summarize the specific troubleshooting process ：

#  View process number pid
top 

#  Confirm the module application corresponding to the process number 
pwdx pid 

#  Find the most time-consuming thread tid
ps -mp pid -o THREAD,tid,time 

#  Turn into 16 Base number 
printf "%x\n" tid

#  The stack finds the specified thread , View the specific running status of the thread 
jstack pid | grep tid -A 30