2. in depth tidb: entry code analysis and debugging tidb

This article is based on TiDB release-5.1 Analyze , Need to use Go 1.16 Later versions

Blog address ：https://www.luozhiyun.com/archives/589

Start up and commissioning

Actually TiDB The debugging of is very simple , What I use here is TiDB release-5.1, So you need to Go The version of is updated to 1.16 after .main The function is in tidb-server Inside the package , Just run it directly . In order to ensure the unity of the environment , I use it Linux Environment .

If you want to debug your own code , It only needs ：

1. install mysql client ;yum install mysql
2. start-up TiDB tidb-server Inside the package main function ;
3. start-up mysql client ;

tidb The default port is 4000 , Account number is root , Library we choose test

mysql -h 127.0.0.1 -P 4000 -u root -D test

1. Breakpoints in the corresponding logic ;

For example, we need to see insert Execution logic , First you need to create a table ：

CREATE TABLE t (

id VARCHAR(31),

name VARCHAR(50),

age int,

key id_idx (id)

);

Break point where logic is inserted ：

image-20210718164645543

Then execute the instrumentation

INSERT INTO t VALUES ("pingcap001", "pingcap", 3);

from main Function to

Learned how to debug TiDB after , Let's see below. TiDB Of main Functions perform logic , It's in tidb-server Under the bag ：

func main() {
	...
	//  register store
	registerStores()
	//  register prometheus Monitoring item 
	registerMetrics()
	//  Setting global  config
	config.InitializeConfig(*configPath, *configCheck, *configStrict, overrideConfig)
	if config.GetGlobalConfig().OOMUseTmpStorage {
		config.GetGlobalConfig().UpdateTempStoragePath()
		err := disk.InitializeTempDir()
		terror.MustNil(err)
		checkTempStorageQuota()
	}
	setGlobalVars()
	//  Set up CPU Affinity 
	setCPUAffinity()
	// Configure the system log
	setupLog()
	//  Regularly check whether the heap memory exceeds the limit 
	setHeapProfileTracker()
	// Register the distributed system tracking chain  jaeger
	setupTracing() // Should before createServer and after setup config.
	printInfo()
	//  Set up binlog Information 
	setupBinlogClient()
	//  Configure monitoring 
	setupMetrics()
	storage, dom := createStoreAndDomain()
	//  establish TiDB server
	svr := createServer(storage, dom)
	//  Set graceful shutdown 
	exited := make(chan struct{})
	signal.SetupSignalHandler(func(graceful bool) {
		svr.Close()
		cleanup(svr, storage, dom, graceful)
		close(exited)
	})
	topsql.SetupTopSQL()
	// Start the service 
	terror.MustNil(svr.Run())
	<-exited
    //  Brush the log disk 
	syncLog()
}

From the above main Method can be seen that it mainly loads configuration items , Then set the configuration information . From the above information configuration , There are a few points that I think can be used for reference in our usual projects , One is the regular detection heap memory detection , The other is graceful downtime .

Detect heap memory detection

The implementation logic of heap memory detection is in setHeapProfileTracker In the method ：

func setHeapProfileTracker() {
	c := config.GetGlobalConfig()
	//  Default 1 minute 
	d := parseDuration(c.Performance.MemProfileInterval)
	//  Run asynchronously 
	go profile.HeapProfileForGlobalMemTracker(d)
}

func HeapProfileForGlobalMemTracker(d time.Duration) {
	log.Info("Mem Profile Tracker started")
	//  Set up  ticker  by 1 minute 
	t := time.NewTicker(d)
	defer t.Stop()
	for {
		<-t.C
        //  adopt  pprof  Get heap memory usage 
		err := heapProfileForGlobalMemTracker()
		if err != nil {
			log.Warn("profile memory into tracker failed", zap.Error(err))
		}
	}
}

As you can see from the above code setHeapProfileTracker It actually starts a Goroutine Asynchronous timing ticker （ Those who are not familiar with the principle of timers can read this article ：https://www.luozhiyun.com/archives/458 ） perform heapProfileForGlobalMemTracker Function by pprof Get heap memory usage .

func heapProfileForGlobalMemTracker() error {
	//  call  pprof  Get heap memory usage 
	bytes, err := col.getFuncMemUsage(kvcache.ProfileName)
	if err != nil {
		return err
	}
	defer func() {
		if p := recover(); p != nil {
			log.Error("GlobalLRUMemUsageTracker meet panic", zap.Any("panic", p), zap.Stack("stack"))
		}
	}()
	//  Put memory in  cache  in 
	kvcache.GlobalLRUMemUsageTracker.ReplaceBytesUsed(bytes)
	return nil
}

heapProfileForGlobalMemTracker By calling pprof Get heap memory usage , Then pass the obtained information to GlobalLRUMemUsageTracker, What's more interesting here is ,GlobalLRUMemUsageTracker yes Tracker Implementation class of , I'll track it Tracker Memory usage of the whole link , If the threshold is reached , Then it will trigger Father Tracker Of hook, Throw out panic abnormal .

tracker

Elegant shutdown

Elegant downtime is more common in projects ,TiDB Will be called at startup SetupSignalHandler Function to perform the corresponding signal listening ：

func SetupSignalHandler(shutdownFunc func(bool)) { 
	closeSignalChan := make(chan os.Signal, 1)
	signal.Notify(closeSignalChan,
		syscall.SIGHUP,
		syscall.SIGINT,
		syscall.SIGTERM,
		syscall.SIGQUIT)

	go func() {
		sig := <-closeSignalChan
		logutil.BgLogger().Info("got signal to exit", zap.Stringer("signal", sig))
		shutdownFunc(sig == syscall.SIGQUIT)
	}()
}

When the supervisor hears SIGHUP 、SIGINT、SIGTERM、SIGQUIT When it's a signal , Will execute the incoming shutdownFunc function ：

	...
	signal.SetupSignalHandler(func(graceful bool) {
		svr.Close()
		cleanup(svr, storage, dom, graceful)
		close(exited)
	})
	...

The incoming to SetupSignalHandler The functions in will first execute server The closing of the ,graceful When the Commissioner heard SIGQUIT When the signal is true, And then call cleanup Perform cleanup operations .

func cleanup(svr *server.Server, storage kv.Storage, dom *domain.Domain, graceful bool) {
	//  Whether it is graceful shutdown 
	if graceful {
		// Elegant shutdown 
		svr.GracefulDown(context.Background(), nil)
	} else {
		//  Try graceful downtime 
		svr.TryGracefulDown()
	}
	//  Clean up all plug-in resources 
	plugin.Shutdown(context.Background()) 
	closeDomainAndStorage(storage, dom)
	disk.CleanUp()
	topsql.Close()
}

cleanup Inside, the connection will be cleaned 、 plug-in unit 、 Disk and shutdown tikv Resources, etc . If graceful yes true, Then call GracefulDown Cycle through idle connections , Until the number of connections is 0; If it is false, Then call TryGracefulDown Clean up the connection , If connected to 15 If the cleaning is not completed within seconds, it will be forced to clean .

Start the service

The process of starting a service is actually related to net/http Of server Very similar . The entrance main At the bottom of the function , adopt server Of Run Method start up ：

func (s *Server) Run() error {
	metrics.ServerEventCounter.WithLabelValues(metrics.EventStart).Inc()
	s.reportConfig()
 
	//  Configure routing information 
	if s.cfg.Status.ReportStatus {
		s.startStatusHTTP()
	}
	for {
		//  Listen for client requests 
		conn, err := s.listener.Accept()
		if err != nil {
			...
		}
		//  establish connection
		clientConn := s.newConn(conn)
 
		//  Handle connection request 
		go s.onConn(clientConn)
	}
}

Run Method here leaves the main logic ：

1. Configure routing information ;
2. monitor connection;
3. by connection Create a separate Goroutine To deal with .

server

The obtained connection will then call connection Of Run Method connection The data of , Then call to connection Of dispatch Method to do request logic forwarding processing .

func (cc *clientConn) dispatch(ctx context.Context, data []byte) error {
	...
	//  Executed command 
	cmd := data[0]
	//  Command corresponding parameters 
	data = data[1:]
	...
	//  take []byte  To  string
	dataStr := string(hack.String(data))
	//  according to  cmd  Select the corresponding execution logic  
	switch cmd {
	case mysql.ComSleep: 
	case mysql.ComQuit: 
	case mysql.ComInitDB:  
	//  most sql  Will follow this logic 
	//  Include add delete change check 
	case mysql.ComQuery:
		if len(data) > 0 && data[len(data)-1] == 0 {
			data = data[:len(data)-1]
			dataStr = string(hack.String(data))
		}
		return cc.handleQuery(ctx, dataStr)
	case mysql.ComFieldList: 
	case mysql.ComRefresh: 
	case mysql.ComShutdown:  
	case mysql.ComStatistics: 
	case mysql.ComPing: 
	case mysql.ComChangeUser:
	...
	// ComEnd
	default:
		return mysql.NewErrf(mysql.ErrUnknown, "command %d not supported now", nil, cmd)
	}
}

dispatch You will get the passed in array , first byte Is the command type , The following is to execute the command , If we insert a insert sentence ：

INSERT INTO t VALUES ("pingcap001", "pingcap", 3);

In this sentence cmd by 3 ,data by INSERT INTO t VALUES ("pingcap001", "pingcap", 3);.

And then according to cmd stay switch Find the corresponding execution logic in the judgment and carry out corresponding processing .

It's important to note that here mysql.ComQuery This branch actually includes addition, deletion, modification and query , You can have a look by yourself .

summary

This article is actually very simple , It mainly talks about how to configure the environment for corresponding debug, Then there is the introduction main What does the method mainly do , And what we can learn from it .

about TiDB We can also refer to the articles written in the previous several times ：《 Let's make it clear Go Language HTTP Standard library 》 Let's take a look at the same server ,TiDB Why do you want to realize one by yourself .

Reference

https://zhuanlan.zhihu.com/p/163607256

https://www.qikqiak.com/post/use-vscode-remote-dev-debug/

https://zh.wikipedia.org/wiki/Unix%E4%BF%A1%E5%8F%B7

Sweep code _ Search for syndication patterns - White version 1