Summary of yarn Explorer

1. From a systematic point of view yarn Mainly by ResourceManager and NodeManager form ,ResourceManager Mainly responsible for resource scheduling ,NodeManager Mainly responsible for operation Container Containers .
2. From application ( such as MapReduce perhaps Spark) From the perspective of yarn It mainly consists of three parts , Namely ResourceManager,ApplicationMaster and NodeManager
3. When client The client submits the application to ResourceManager after ,ResourceManager Will use the client settings cpu And memory parameters start first ApplicationMaster Applications , meanwhile ApplicationMaster And ResourceMaster Establish a heartbeat channel between ,ApplicationMaster We will inform you through this channel ResourceMaster Your survival , Normally , An application can only have one ApplicationMaster, But if there is a network partition , It may also consist of multiple ApplicationMaster, It is also known as brain fissure , This abnormal condition will cause multiple programs to write the same shared data , Cause data corruption , We need to try our best to avoid this kind of situation
4.NodeManager Will monitor Container Operation of , Prevent it from using more memory and cpu resources , When that happens ,NodeManager Will directly kill Drop the corresponding container .
5. When the application runs ,NodeManager Local logs will be aggregated and stored in hdfs In a public directory ( Configurable ), In order to persist logs and provide users with access .
6. When the application ends , Under normal circumstances ApplicationMaster Will stop all containers and move to ResourceManager Send end message , When ResourceManager When you receive the message that the application ends , He will tell everyone NodeManager The node sends a request to confirm all that belong to the application container Have been killed , If there are still some container If you were in a state of suspended animation ,NodeManager received RM After the news , Will be forcibly killed and cleaned up container.
7.ApplicationMaster towards ResourceManager The way to request resources is as follows ： Suppose there are three computer rooms M1,M2,M3, There are three machines in each machine room , Namely A,B,C, Now there are two copies of data stored M1A, and M2B On , Two... Are needed at this time Container To run the task , When ApplicationMaster towards ResourceManager When applying for resources , Will tell RM Priority in M1A and M2B Apply for a container on the machine , And tell RM If resources are not enough , Then it can be on the rack A And rack B The above application Container, At the same time, continue to tell RM If rack A And rack B The resources on are still insufficient , Then you can allocate any two on any computer room Container to ApplicationMaster Use , Determine according to the location of the data Container The priority relationship of the assigned location is ApplicationMaster The responsibility of the .
8. RM and NM Will maintain a channel ,NM Will tell through this channel RM Current resource usage , such RM There will be an overall overview of the current resource usage of the cluster , In order to RM Allocate resources
9. When NodeManager function Container when , He will first include the remote resources jar file , Download the configuration file to the local directory , These resource types include application wide resources , User wide resources ,public Range ( All users are available ) Resources for , When to start Container The required resources are loaded from the local directory
10. about yarn Come on , Each application can be submitted to a different priority queue when it is submitted , We can do it in yarn Run through yarn rmadmin -- Command to add a new priority queue or remove the application from the priority queue A Adjust to priority queue B in