Flynk deployment mode summary

be based on flink-1.15.0

Independent of external resource management framework

flink local Deployment mode

This deployment mode is mainly used to test

flink standalone Deployment mode

This deployment mode , from flink Be responsible for resource management and allocation

flink application Deployment mode

This deployment mode ,main Function in jobmanager Run and build job graph, Reduce client stress

Deployment way ：flink Deployment mode （ One ）- standalone and application

flink on yarn

Session Mode on yarn

This deployment mode , share jobmanager and taskmanager, be-all job All in one runtime Run in .

advantage ：
The only way to start a cluster is jobmanager, Submit job Just to go yarn Apply for resource launch taskmanager, Task completion automatically releases taskmanager, Good resource scalability .
High resource utilization .
shortcoming ：
Poor isolation of resources

Per-Job Mode on yarn

exclusive jobmanager and taskmanager, For every one job Start one alone runtime

advantage ：
Resources are fully isolated
Inferiority ：
Resources are relatively wasteful

Application Mode on yarn #

application Of main() Running on the jobmanager On , Not on the client side . every last application Corresponding to one runtime,application Can contain multiple job.

advantage ：
job graph stay flink Generated within the cluster , It can reduce the pressure on the client
application A more reasonable resource isolation strategy is implemented
shortcoming ：
new function , It has not been verified on a large scale in production

Deployment way ：flink Deployment mode （ Two ）- yarn Three deployment patterns

flink on k8s

standalone k8s session mode

Pre built Flink colony , And the cluster is running for a long time , But it can't expand or shrink automatically . User pass client Submit the job to the running JobManager, and JobManager Assign tasks to running TaskManager.

advantage ：

• Flink The cluster is started and run in advance , When a user submits a job , Jobs can be immediately assigned to TaskManager, That is, the job starts fast .

shortcoming ：

• Low resource utilization , Determine in advance TaskManager Number , If the job requires less resources , Then a large number TaskManager In idle state , conversely TaskManager Insufficient resources .
• Poor job isolation , The task of multiple jobs has resource competition , interact . If a job exception causes TaskManager Hang up , The TaskManager All jobs on will be restarted .

Deployment way ：flink Deployment mode （ 3、 ... and ）- standalone k8s session Deployment mode

standalone k8s application mode

Each job has an exclusive Flink colony , When the homework is finished , Cluster recycled .
The application image needs to be built in advance ,Flink The image needs to contain jobs, i.e Application Rely on the Class.
When starting a job, you need to specify Main Function entry class .

advantage ：

• A job monopolizes a cluster , Good isolation of operation .
  shortcoming ：
• The application image needs to be built in advance , More trouble
• Low resource utilization , It needs to be determined in advance TaskManager Number , If the job requires less resources , Then a large number TaskManager In idle state , conversely TaskManager Insufficient resources . meanwhile JobManager It can't be reused .

Deployment way ：flink Deployment mode （ Four ）- standalone k8s application Deployment mode

native k8s session mode#

similar 【standalone k8s session mode】 Pattern , You need to pre build JobManager. The difference is that users use Flink Client towards JobManager After submitting the assignment ,
According to the needs of the operation Slot Number ,JobManager Direct to Kubernetes apply TaskManager resources , Finally, submit the homework to TaskManager On .

advantage ：

• TaskManager Is real-time 、 On demand creation , More efficient use of resources .
• Good resource scalability ,job After that , Idle TaskManager Will be destroyed to free up resources .

shortcoming ：

• job It takes a long time to really run , Because of the need to wait TaskManager establish .

Deployment way ：flink Deployment mode （ 5、 ... and ）- native k8s session Deployment mode

native k8s application mode #

similar 【standalone k8s application mode】 Pattern , Each job has an exclusive Flink colony , When the homework is finished , Clusters will also be recycled .
The difference is Native characteristic , namely Flink Direct and Kubernetes Communicate and request resources on demand , There is no need for the user to specify TaskManager The amount of resources .

advantage ：

• A job monopolizes a cluster , Good isolation of operation .
• The utilization rate of resources is relatively high , Apply on demand JobManager and TaskManager.

shortcoming ：

• The application image needs to be built in advance , More trouble
• A job monopolizes a cluster ,JobManager It can't be reused .
• The job starts slowly , After the assignment is submitted , Just started to create JobManager and TaskManager.

Deployment way ：flink Deployment mode （ 5、 ... and ）- native k8s application Deployment mode

summary

• If it is yarn Do resource management , Recommended 【Application Mode on yarn】 , At the same time combined with yarn Rely on pre upload settings .

• If it is k8s Do resource management , Recommended 【native k8s session mode】 perhaps 【native k8s application mode】, Better resource utilization and resource scalability . If you use session Pattern ,main() Function on the client side job graph, Increase client pressure ; If you use application Pattern , It may be necessary to build application images frequently . This needs to be comprehensively evaluated according to specific use scenarios .