48 pictures | teach you the performance monitoring, pressure testing and tuning of microservices by hand

This is Wukong's first 85  Original articles

author | Wukong chat structure

source |  Wukong chat structure （ID：PassJava666）

Please contact authorization for reprint （ WeChat ID：PassJava）

Main contents of this paper

The open source project of this article ：https://github.com/Jackson0714/PassJava-Platform

This article has been included in ：www.passjava.cn

One 、 What is stress testing

1.1、 Big white explanation

• What is performance pressure measurement ： It's about looking at the present Software and Hardware In the environment , What the system can withstand Maximum load , And help find out what the system bottleneck Where .

• The purpose of performance pressure measurement ： For the system online processing capacity and stability Stay in a Standard range Inside , Know yourself and your enemy , you can fight a hundred battles with no danger of defeat . You can also find memory leaks 、 Concurrency and synchronization .

1.2、 Performance indicators

• RepsonseTime - RT： response time , The user initiates a request from the client to start the calculation , When the client receives the response from the server, it ends , The time spent in the whole process .

• Hits Per Second - HPS： User clicks per second , That is, the number of requests sent to the background per second .

• QPS： The number of queries processed by the system per second .

• MaxRT： Maximum response time , It refers to the maximum time from the user's request to the server's response .

• MiniRT： Minimum response time , It refers to the minimum time from the user's request to the server's response .

• 90% response time ： Sort the response times of all users in ascending order , take 90 % The location of .

• Performance testing concerns ：

  • throughput ： The number of requests that the system can handle per second 、 Number of tasks .

  • response time ： The time taken by a service to process a request or a task .

  • Error rate ： The proportion of requests that result in fault in a batch of requests .

Two 、Jmeter Pressure measuring tool

1、Jmeter Tools

• Download and install Jmeter Tools

Download address ：

https://jmeter.apache.org/download_jmeter.cgi

The version I downloaded is apache-jmeter-5.3

• function JMeter Program

Open the batch file ：\apache-jmeter-5.3\bin\jmeter.bat. As shown in the figure below ：

• Add thread group , As shown in the figure below ：

Add thread group

• 1s Start inside 200 Threads , cycles 100 Time .2 w A request . As shown in the figure below ：

• test HTTP request , As shown in the figure below ：

Configure the protocol to be tested 、 Server address 、 Port number . The configuration information is as follows ：

agreement ： Use http agreement .

Server name or IP: www.baidu.com ( Just to demonstrate ).

Port number ：80 port .

As shown in the figure below ：

• Add view result tree 、 Summary reports and aggregate reports . As shown in the figure below ：

• Start the stress test .

  Click the play button to start . Note that before starting, you need to set the parameter configuration and HTTP Requested configuration . As shown in the figure below ：

• View the results of each request , As shown in the figure below ：

• View the summary report .

  Mainly concerned with average and throughput .

  200 Threads , Call per thread 100 Time , in total 2 w Time , As you can see, the sample column in the table below is also 2 w, The time taken by the request is 151 ms, Throughput is 880 Requests per second . As shown in the figure below ：

• View aggregation reports .

It's mainly about the median and 90% Percentile ,

Median yes 59 ms, Explain that the response time for most requests is 59 ms.

90 % Request It's all in 271 ms It's done in response .

abnormal 0.41% explain 2 w One of the requests is 82 A request exception （20000 * 0.0041 = 82 ）.

throughput 880.2/sec That Baidu can handle every second 880 A request , Average performance （ It may be related to the performance of the local machine ）.

As shown in the figure below ：

• Look at the summary chart .

When looking at the summary chart , You need to check the information you want to view first , As shown in the figure below ：

Then look at the graph summary ：

You can see that the selected columns are shown in different colors in the chart , For example, the green bar is 90 % Percentile .

So let's test that out PassJava（ The best is the best ） Manage the performance of the background , The throughput is close to 2000/s. As shown in the figure below ：

3、 ... and 、 Performance monitoring jconsole

jconsole and jvisualvm yes Java JDK Two of my little tools , Used to monitor memory leaks 、 Tracking garbage collection 、 Memory at execution time 、 Yes CPU Analyze 、 Thread analysis . Can be started from the command line , And it can monitor local and remote applications . and jvisualvm It's an upgraded version of jconsole. So let's see jconsole Use .

First use cmd Command line way to start jconsole.

start-up jconsole‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Command line start jconsole

Which application monitor chooses

And then choose passjava Project question service .

choice passjava-question Microservices

Corresponding to the following micro service ：passjava-question.

Corresponding passjava-question Microservices

overview

From the monitoring interface there are 6 Menu , The first thing you see is the overview function , It has heap memory usage 、 Number of threads 、 Class usage 、CPU Occupancy rate , It's all represented by a trend chart , A convenient overview of current performance . Be careful ： The monitoring is real-time .

overview

Memory

Here's how memory is used , As you can see from the figure below, there is a drop-down box , You can choose different memory dimensions , Then the icons and bar charts below will show the different dimensions according to the selected dimension .

Memory

Here's how threads are used , You can see the peak number of threads and the total number of active threads , The peak seen so far is 59, The number of active threads is 57. In the second half, you can see which threads , And the stack information of the thread , Very detailed .

Thread usage

class

Here is the loading and unloading of the class , The total number of loaded classes is 10679, The unloaded class is 1 individual , So the total number of classes that have been added is 10679 - 1 = 10678 individual .

Class loading and unloading

VM Summary

Let's see VM（ virtual machine ） The situation of . As shown in the figure below , You can see the virtual machine , Threads 、 class 、 Heap profile , as well as VM Parameters of , Is it convenient ~

VM Summary

MBean Information

Let's take a look at MBean Information . about MBean, Maybe many students don't know what it is , Here's an explanation ：

MBean It's a kind of normative JavaBean, Through the integration and implementation of a set of standards Bean Interface , This is called MBean.MBean What can be used for ？ That is, there can be a set of JDK Level external service interface . such as , You wrote a JVM Allows state assisted queries Bean, You want someone else to download one Jconsole You can see your masterpiece . Then you can consider using MBean Norms to achieve . A lot of garbage collector algorithms Bean That's it （ That's the class sun.management.MemoryImpl）.

MBean Information

Four 、 Performance monitoring jvisualvm

jvisualvm Than jconsole More powerful , The interface is more informative . You can also install plug-ins , That's great. .

start-up jvisualvm And Overview

Starting mode and jconsole equally , through cmd Command line start . Or choose passjava-question Microservices , Then select the first menu bar ： summary . You can see JVM Version of , Start parameters and other information .

start-up jvisualvm

monitor

monitor CPU、 Pile up 、 class 、 Thread situation . The overall effect is better than jconsole More beautiful .

monitor

Threads

Check the thread again . You can see that there is 5 Threads in two states ：

• function ： Running thread .

• Sleep ： Dormant threads .

• wait for ： Threads waiting for execution .

• Resident ： Idle threads in threads .

• monitor ： Blocked threads , Waiting for lock .

A sampler

In addition, we can also use a sampler to CPU Or memory . As shown in the figure below , Sampling memory .

Sampling

Use of plug-ins

install Visual GC plug-in unit

Installation steps ： Tools -> plug-in unit -> Available plug-ins ->Visual GC-> install . After installation , Restart to use the plug-in function .

After installation , You can see that

Visual GC plug-in unit

The figure below shows the real-time monitoring of garbage collection .

Visual GC

5、 ... and 、 Performance test of gateway

Now I'd like to Passjava Systematic question Microservice interface for a pressure test , How to do it ？

So let's look at this first passjava What's the architecture of , As shown in the figure below ：

The client is divided into mobile terminal and PC End ,http Request to go first API Gateway, And then forward it to question Microservices . It involves middleware ：Gateway gateway .

We come to the right Gateway The gateway is stress tested .

The port number of the gateway is 8060, Let's configure JMeter. As shown in the figure below ：

Configure send per second 200 A request , It's going to cycle all the time , Until the pressure measurement is stopped manually . As shown in the figure below ：

You can see the results of the execution , Throughput in 2422 Per second , It's still higher .

throughput ：2422/s .

90% response time ：142 millisecond .

99% response time ：281 millisecond .

Let's look at garbage collection ,Eden It's time for garbage collection in the district 2.7 s, It's too long , Let's see how this can be optimized .

The general optimization direction is to increase the new generation heap memory configuration .

6、 ... and 、 Performance testing of microservices

According to the architecture diagram above , We know that client requests go through Gateway It was forwarded once , If we want to see how to test the performance of microservices alone ？ Let me show you how to test passjava-question The performance of microservices .

The first thing you need to do is passjava-question Add a test method to the microservice ：

The test method

There are two ways to test this api Is it correct to add .

The first is to use postman Test whether the request can respond correctly , return “test” The response is correct .

test api Whether it can respond correctly

The second is to test through a browser . After entering the following link in the browser address bar , enter , See if the browser window displays “test”, If yes, the response is correct .

And then we need to use Jmeter Pressure testing tool to test this microservice api What's the performance of .

Independent pressure measurement micro service api Result

throughput ：3542/s .

90% response time ：100 millisecond .

99% response time ：152 millisecond .

7、 ... and 、 To gateway + Performance testing of microservices

If we want to test the performance of the whole request link, what should we do ？

First, the request needs to go through the gateway , Then the gateway forwards it to the microservice . In the previous post , I've configured the gateway , So to forward through the gateway to test request , You just need to modify the request path a little , As shown below ：

http://localhost:8060/api/question/v1/admin/question/test

Then enter the URL in the browser , return “test” That means the response is correct .

And then we still use Jmeter Under the pressure test tool test test api Performance of . The test results are shown in the following figure ：

gateway + Stress test results of microservice

As you can see from the results ：

throughput ：982/s .

90% response time ：437 millisecond .

99% response time ：790 millisecond .

Here's a horizontal comparison ：

Horizontal contrast

Explain microservices api After one forwarding through the gateway , Performance is at least halved . It can be concluded that ： The more middleware , The greater the performance loss , Most of the losses are due to network interaction . The delay of network can be reduced by improving the quality of network communication .

8、 ... and 、 Optimize the database query

In general , The performance problem is more about the time-consuming of querying the database in the business . Next, let's look at how to optimize the data query .

Here is a list of query questions api： By question type type Field filter problem list .api Path as follows ：

http://localhost:11000/question/v1/admin/question/list?type=5

This api The code for is as follows , It's easy to understand .

Query question list api

Let's add some test code ： Count the time to query the database . As shown below ：

Time consuming Statistics

And then restart passjava-question service , Test this again api, Time consuming 43 ms

How to optimize the query ？ It's easy to think of , Let's try adding question The effect of adding index to table . to type Field plus normal index , As shown in the figure below ：

Add index

Let's look at the time-consuming situation after indexing ： Time consuming 18 ms, It's better than before 43 ms Much faster. .

After indexing

Nine 、 Optimize garbage collection

We can go through jvisulavm Tools to see garbage collection ,Eden It happens all the time GC, short time （1 minute ） The Communist Party created 480 Time stop the world. In addition, it can be seen from the pressure measuring tool , Throughput for 275/s.

as a result of Eden The memory allocation of the area is too small , Only 32 M, Let's make it bigger .

32M Eden There is frequent garbage collection in this area

increase Eden Area size

By means of IDEA The following parameters are configured in the tool , Adjust the maximum heap memory to 1024 M, The new generation memory is 512 M.

-Xmx1024m -Xms1024m -Xmn512m

Then it can be observed that in a very long and short time （1 minute ） I only did 92 Garbage collection , It means that the frequency of garbage collection is reduced . Application performance has also improved . In addition, it can be seen from the pressure measuring tool , Throughput for 347/s, The throughput has also been greatly improved .

Ten 、 summary

In this paper, through the pressure measurement tool Jmeter How to implement decompression test , Then use the performance monitoring tool jconsole and jvisualvm To monitor Java Application performance , And how to use tools to optimize open source projects passjava Performance of , And very detailed description of each step and the implementation results , By way of comparison , Know more clearly how to do performance optimization .

Here are the general optimization methods for system performance ：

• When there are many middleware , Optimize the quality of network communication .

• Database query time consuming , The query needs to be optimized , Like adding indexes .

• The rendering speed of the template , You can set the template cache .

• Acquisition of static resources , Can pass Nginx Separation of motion and static to solve .（ Let's talk about it next time ）

• Too many logs , We need to reduce unnecessary fighting log operation .

The shoulders of giants ：

https://blog.csdn.net/u010833547/article/details/92806510
https://www.bilibili.com/video/BV1np4y1C7Yf
https://github.com/Jackson0714/PassJava-Platform
www.passjava.cn

- END -

Author's brief introduction ： The wu is empty ,8 Years of experience in Internet development and architecture , Use stories to explain 、 Architecture design 、Java The core technology .《JVM Performance optimization practice 》 Columnist , Open source 《Spring Cloud actual combat PassJava》 project , Independently developed a PMP Brush topic small program .

 

Previous recommendation

 

 

Han Xin's big move ： Consistent Hashing

Use Taijiquan to teach distributed theory , It is really comfortable ！

Talk about distributed algorithms in Three Kingdoms , Comfortable, right ？

Ali's final face ： How to use it? UDP Realization TCP？

5w word | 172 chart | Super Saiya Spring Cloud actual combat

I'm Wukong , Try to be strong , Become a super Saiya ！

This article is from WeChat official account. - Wukong chat structure （PassJava666）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .