A troubleshooting of golang memory leak

background

Recently in use golang Develop a content recommendation project , It was found that the vessel would restart after a period of time before the pressure test was planned , When checking the memory of the machine, it is found that the memory has been rising since startup , Then restart after reaching the maximum available memory threshold of a container . Here's the picture ：

image.png

From the above chart, it can be concluded that , Memory leaks .

The screening process

One thing to note is that golang Is based on goroutine On schedule , therefore goland Ninety percent of memory leaks come from goroutine Memory leak , We just need to keep an eye on goroutine The most places , You can basically find the source of the memory leak .

I use the go Of pprof The tool has been debugged and checked , First, introduce this component into the project , And desensitize all the codes that may leak business information .

package main

import (
	"fmt"
	"net/http"
	_ "net/http/pprof"
)

func main() {
   	//  Ignore other codes 
//pprof server
http.ListenAndServe("localhost:8080", nil)
  	//  Ignore other codes 
}

First, let's take a look at the situation when there is no traffic at startup ,CPU And memory allocation ：

image.png

Let's go in goroutine You can see a total of 21 individual goroutine, The most 5 One is used to run timed tasks . At present, the project is normal .

image.png

Because my service has multiple external interfaces , So I turn off the other interface routes first , Then use the pressure measuring tool for pressure measurement （ What I'm using here is go-stress-testing)

image.png

Then let's look at pprof The indicators of

Find out goroutine It's skyrocketing , Click in and find that there is 1001 individual redis pool object , It's normal for us redis The connection pool will only have 1 One of the .

My code is like this

package utils

import (
	"fmt"
	"github.com/go-redis/redis/v8"
	"time"
)

func GetRedisClient() *redis.Client {
	redisCli := redis.NewClient(&redis.Options{
		Addr:         "localhost:6379",
		Password:     "", // no password set
		DB:           0,  // use default DB
		PoolSize:     20,
		MinIdleConns: 10,
		IdleTimeout:  3 * time.Minute,
	})
	if redisCli != nil {
		return redisCli
	} else {
		fmt.Sprint("redis client is nil")
		return nil
	}
}

Use redisClient The local code for is

func (s baseService) Multiply(ctx context.Context, in *pb.MultiplyRequest) (errCode int32, resp *pb.MultiplyResponse, err error) {
	//  Ignore other codes 
	redisCli := utils.GetRedisClient()
	userFeat, err := redisCli.Get(ctx, "user_feat:" + uid).Result()
	if err != nil {
		return comn.SUCC.Code, &pb.MultiplyResponse{
			Code: comn.ErrRequestParamIllegal.Code,
			Msg:  comn.ErrRequestParamIllegal.Msg,
		}, nil
	}
	//  Ignore other codes 
}

Found it. , original go-redis Under bag redis.Client It's a Connect pool objects Not a simple one Client connection

Knowing the problem, we just need to redis.Client Initialize as a global object , Each time you need to use it, you can directly reuse the previous connection pool .

The code is adjusted as follows ：

import (
	"fmt"
	"github.com/go-redis/redis/v8"
	"time"
)

var RedisClient *redis.Client

func init() {
	RedisClient = redis.NewClient(&redis.Options{
		Addr:         "localhost:6379",
		Password:     "", // no password set
		DB:           0,  // use default DB
		PoolSize:     20,
		MinIdleConns: 10,
		IdleTimeout:  3 * time.Minute,
	})
	fmt.Sprint(RedisClient)
}

Reuse directly each time a request comes redis.Client Just go ：

func (s baseService) Multiply(ctx context.Context, in *pb.MultiplyRequest) (errCode int32, resp *pb.MultiplyResponse, err error) {
	//  Ignore other codes 
	userFeat, err := utils.RedisClient.Get(ctx, "user_feat:" + uid).Result()
	if err != nil {
		return comn.SUCC.Code, &pb.MultiplyResponse{
			Code: comn.ErrRequestParamIllegal.Code,
			Msg:  comn.ErrRequestParamIllegal.Msg,
		}, nil
	}
	//  Ignore other codes 
}

Restart , Pressure test again ：

image.png

The discovery is basically goroutine The quantity of is kept at a reasonable value .

image.png

Then release the code to the test container , After a period of time, the memory usage in the monitor is basically normal .

image.png

summary

• golang Most memory leaks are caused by goroutine Caused by leakage , So we should first focus on the overall goroutine Number .
• image redis,mysql This relatively scarce resource can be used Connection pool But pay attention to the size of the control pool , And the timeout for each connection , Pay attention to timely release after use . and go Yes redis also mysql Both have built-in connection pools , You must find out what the initialized instance is .