Good fat man takes you to learn Flink series -flink source code analysis episode I standalone startup script analysis

A good fat man takes you to learn Flink：Flink The first episode of source code analysis Standalone Start script analysis

1、 Preface

Hello, friends , Next I will bring you a complete set of Flink The source code learning tutorial will include several knowledge points ：

1. Flink Learning the kernel process
2. Study Flink Learning from this top-level open source framework
3. At the same time, we will provide you with some small issue, Can it be commiter It depends on everyone .

2、 Content

1. start-cluster.sh

At this time, all leaders , You can open your own flink Source code , Or download it Flink Packages and centos above Flink In software bin Catalog .

stay flink Source code , The project where the script is located is flink-dist Under the flink-bin Catalog , At this point we are analyzing start-cluster.

Many friends must say why not watch yarn Submit , Want to see Standalone How to start the cluster , because Standalone Every computing framework exists , for example yarn And so on are all done for compatibility ,standalone Only in this way can we truly reflect flink Some usage of some internal core frameworks （ Because it is relatively simple and easy to analyze ...）, Let's analyze , Start script , In fact, many students are analyzing the source code of a framework , Often in order to find the entrance and trouble , At work , This is also the case , For example, look at a not springboot The program , Just look at his main Method .

If it is springboot It depends on the display effect of his page ,debug Interface , This will make you familiar with the source code very quickly , After that, we will look at some internal asynchronous 、 Some methods of timing .

So let's analyze the script .

First, he will execute a very important script config.sh

This script is really important , It is an acquisition configuration , And important scripts to get some parameters , In order to be compatible with Huawei cloud in our company's project , This script is often modified , Add something to this script .

You can see

#  Is it highly available 
if [[ $HIGH_AVAILABILITY == "zookeeper" ]]; then
    #  obtain mater The address of 
    readMasters
    #  Remote execution 
    ssh -n $FLINK_SSH_OPTS $master -- "nohup /bin/bash -l \"${FLINK_BIN_DIR}/jobmanager.sh\" start ${master} ${webuiport} &"
        fi
    done

At this time, many students got confused readMasters What is this ？ In fact, does this affect us to watch the script , No effect , I don't really understand shell, But I can probably understand This is a shell Methods , According to the method name He is reading master The address of . then ssh To this master Start the machine jobmanager. Why must it be started on a fixed machine jobmanager, Because he has a commonRpcService, Other components will look for this service. So what must be fixed , You configure that machine to execute on that machine , If the error , Can't access service Of .

Because the passing of all nodes largely depends on the configuration file to generate services , For other components to communicate .

Said so much , Where is this method ？ This method is actually config.sh

Let's assume that we are not currently a highly available cluster , We must be leaving else, So what he called must be "$FLINK_BIN_DIR"/jobmanager.sh start This script , Let's go and have a look .

2. Jobmanager Start class analysis

At this point we open , This script , We see the bottom , to glance at What did he write

What we have introduced is start, I'm sure I won't go here at this time if, Is take the else, That is to say

"${FLINK_BIN_DIR}"/flink-daemon.sh $STARTSTOP $ENTRYPOINT "${args[@]}" This command , that $ENTRYPOINT What is it? , It is also defined above

If this script is completely written , It can be written.

flink-daemon.sh start standalonesession "${args[@]}"

We enter flink-daemon.sh It's very simple. We found that

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-N8V6unT4-1655545235858)(https://gitee.com/wuqingzhi128/blogImg/raw/master/image-20220618172152405.png)]

Here is common sense , I don't have to look down , You know jobmanager It must be this kind of walking main Method . start-up JobManager Walking StandaloneApplicationClusterEntryPoint This class main Method

3. TaskManager Start class analysis

We are at this time , We have found ,JobManager The entrance to , But we haven't found it yet Taskmanager Entrance , Let's go back to start-cluster.sh .

Below we can see a method start-up , We found this way , This method is config.sh in .

You can see that it is started locally , Start remotely instead of locally taskmanager, Of course, there may be more than one , That is to say for Cycle to start .

We see the internal call taskmanager.sh start, Let's see taskmanager.sh This script , Don't say , Layer by layer .

We searched and searched , We found this piece , Is it right? JobManager Startup is very similar .

It's like

/flink-daemon.sh start taskexecutor "${ARGS[@]}"

Go back to flink-daemon.sh see , We also found ,Taskmanager The last class to execute is TaskManagerRunner This class of main Method .

At this time, we found all the entrances , The rest is to look at the source code .

3、 summary

1、 We analyzed start-cluster.sh Script for , In this script , He will call... First config.sh Get configuration .

2、 Will be based on JobManager Is it right? Ha To determine whether to execute different branches remotely JobManager

3、 We're sure JobManager The startup class of is StandaloneApplicationClusterEntryPoint

4、 We're sure TaskManager The startup class of is TaskManagerRunner This class .

If the interview asks if you have seen flink Source code , It can be said that the summary of our script analysis , Of course, that's not the point , In the next issue, we will analyze JobManager Start source code of .