Detailed explanation of Flink operation architecture

1. Flink  Program structure

Flink  The basic building blocks of a program are flows and transformations （ Please note that ,Flink  Of  DataSet API  Used in  DataSet  It's also internal flow  ）. conceptually , Flow is （ It may never end ） Data record stream , Transformation is the operation of one or more flows as one or more flows . Input , And produce one or more output streams .

Flink  The application structure is shown in the figure above ：

Source

:  data source ,Flink  On stream processing and batch processing  source  There are about  4  class ： Based on local collection  source、 file-based  source、 Based on network socket  source、 Self defined  source. Self defined  source  Common are  Apache kafka、RabbitMQ  etc. , Of course, you can also define your own  source.

Transformation

： Various operations of data conversion , Yes  

Map / FlatMap / Filter / KeyBy / Reduce / Fold / Aggregations / Window / WindowAll / Union / Window join / Split / Select

etc. , A lot of operation , You can convert the data into the data you want .

Sink

： Receiver ,Flink  Where to send the converted data  , You may need to store it ,Flink  common  Sink  There are about the following categories ： write file 、 Print out 、 write in  socket 、 Self defined  sink . Self defined  sink  Common are  Apache kafka、RabbitMQ、MySQL、ElasticSearch、Apache Cassandra、Hadoop FileSystem  etc. , In the same way, you can define your own  sink.

2. Flink  Parallel data stream

Flink  When the program is executed , Will be mapped to a  Streaming Dataflow, One  Streaming Dataflow  It's a group.  Stream  and  Transformation Operator  Composed of . Start from one or more  Source Operator  Start , End with one or more  Sink Operator.

Flink  Programs are parallel and distributed in nature

, In the process of execution , A stream (stream) Contains one or more stream partitions , And every one  operator  Contains one or more  operator  The subtasks . Operation subtasks are independent of each other , Execute in different threads , Even on different machines or different containers .operator  The number of subtasks is this particular  operator  Parallelism of . Different in the same program  operator  There are different levels of parallelism .

One  Stream  Can be divided into multiple  Stream  The partition , That is to say  Stream Partition. One  Operator  It can also be divided into multiple  Operator Subtask. Such as in the figure above ,Source  Is divided into  Source1  and  Source2, They are  Source  Of  Operator Subtask. every last  Operator Subtask  Are executed independently in different threads . One  Operator  Parallelism of , Is equal to  Operator Subtask  The number of . Upper figure  Source  The parallelism of is  2. And one  Stream  The parallelism of is equal to the  Operator  Parallelism of .

The data is in two  operator  There are two modes of transmission between ：

One to One  Pattern

： Two  operator  When passing in this mode , It will keep the number of data partitions and data sorting ; As shown in the figure above  Source1  To  Map1, It's reserved  Source  Partition properties , And the order of partition element processing .

Redistributing （ Redistribution ） Pattern

： This pattern changes the number of partitions in the data ; Each one  operator subtask  According to the selection  transformation  Send data to different targets  subtasks, such as  keyBy() Will pass  hashcode  Repartition ,broadcast() and  rebalance() Method will be randomly re zoned ;

3. Task  and  Operator chain

Flink  All operations of are called  Operator, When submitting a task, the client will respond to  Operator  Optimize the operation , Can merge  Operator  Will be merged into one  Operator, After the merger  Operator  be called  Operator chain, It's actually an execution chain , Each execution chain will be in  TaskManager  Executed in a separate thread .

4.  Task scheduling and execution

• When  Flink  perform  executor  It will automatically generate... According to the program code  DAG  Data flow diagram ;

• ActorSystem  establish  Actor  Send the data flow diagram to  JobManager  Medium  Actor;

• JobManager  Will continue to receive  TaskManager  The heartbeats of , Thus, effective  TaskManager;

• JobManager  Through the scheduler  TaskManager  Scheduling execution  Task（ stay  Flink  in , The smallest scheduling unit is  task, The corresponding is a thread ）;

• While the program is running ,task  And  task  Data can be transmitted between .

Job Client

：

• The main responsibility is to submit tasks ,  After submission, the process can be ended ,  You can also wait for the result to return ;

• Job Client  No  Flink  The internal part of program execution , But it's the starting point for task execution ;

• Job Client  Responsible for receiving user's program code , Then create the data stream , Submit the data stream to  Job Manager  In order to carry out further .  After execution ,Job Client  Return the result to the user .

JobManager

：

• The main responsibility is to schedule work and coordinate tasks to do checkpoints ;

• At least one in the cluster  master,master  Responsible for scheduling  task, Coordinate  checkpoints  And fault tolerance ;

• High availability settings can have more than one  master, But make sure one is  leader,  The others are  standby;

• Job Manager  contain  Actor System、Scheduler、CheckPoint  Three important components ;

• JobManager  After receiving the task from the client ,  First, an optimized execution plan is generated ,  Reschedule to  TaskManager  In the implementation of .

TaskManager

：

• The main responsibility is to start from  JobManager  Mission reception ,  And deploy and start tasks ,  Receive upstream data and process ;

• Task Manager  Is in  JVM  The work node in one or more threads of ;

• TaskManager  Set it up at the beginning of creation  Slot,  Every  Slot  Can perform a task .

5.  Task slot and slot sharing

Every  TaskManager  It's a  JVM  The process of ,  You can perform one or more subtasks in different threads . In order to control a  worker  How many  task.worker  adopt  task slot  To control （ One  worker  At least one  task slot）.

1)  Task slot

Every  task slot  Express  TaskManager  Having a fixed size subset of resources .

flink  The memory of the process is divided into several  slot  in .

The picture shows  2  individual  TaskManager, Every  TaskManager  Yes  3  individual  slot  Of , Every  slot  occupy  1/3  Of memory .

Memory is divided into different  slot  Then you can get the following benefits :

• TaskManager  The tasks that can be executed concurrently at most are controllable , That's it  3  individual , Because it can't be more than  slot  The number of .

• slot  Have exclusive memory space , This is in a  TaskManager  Multiple different jobs can be run in , Not affected between jobs .

2)  Slot sharing

By default ,Flink  Allow subtasks to share slots , Even if they are subtasks of different tasks , As long as they come from the same assignment . The result is that one slot can hold the entire pipeline of the job . Allowing slot sharing has two main benefits ：

• Just calculate  Job  Medium maximum parallelism （parallelism） Of  task slot, As long as this is satisfied , Other  job  Can also be satisfied .

• More equitable distribution of resources , If you have more free  slot  More tasks can be assigned to it . If there is no task slot sharing in the figure , Low load  Source/Map  etc.  subtask  Will occupy a lot of resources , Windows with higher loads  subtask  There will be a lack of resources .

• With task slot sharing , The basic parallelism can be （base parallelism） from  2  Upgrade to  6. The utilization rate of slotted resources is improved . At the same time, it can also ensure  TaskManager  to  subtask  The distribution of  slot  The scheme is more equitable .