Applicationclientprotocol of yarn source code

Introduction

yarn adopt Protocol Protocol layer to communicate interface layer and core logic layer .

ApplicationClientProtocol The agreement is responsible for Client and ResourceManager Interaction logic of . The main functions are submit/abort jobs（ Submit / Terminate task ） and get information from applications（ Get application information ） as well as get cluster metrics（ Get cluster indicators ）. This article will focus on the main functions

summary

ApplicaitionBaseProtocol

This agreement inherits ApplicaitionBaseProtocol Interface , This is because access to application information , Get container information , management Delegation token（ Delegate flag ） These functions can be reused by other protocols , So we extracted a base Interface

Put aside what the parent interface does ,ApplicationClientProtocol There are only a few functions left

getNewApplication Get a new one ApplicaitonId, And get the RM The maximum resources currently available for allocation

submitApplication Formally submit an application , You need to include the details of the application , for example queue,resource,containerLaunchContext etc.

forceKillApplication Force to kill an application

getClusterMetrics Get cluster indicators

getClusterNodes Get the cluster node report

Three realizations

The three implementations are located in three layers

ApplicationClientProtocolPBClientImpl, At interface layer , Directly be client Interface calls are provided to the user

ClientRMService, At the core logic layer , from RM Realization , Contains all the core interaction logic

ClientRMProtocols, be located Poxy Agent interaction layer , Used to expose the calling interface to the interface layer

Submit the overall task flow

client Request

client call getNewApplication Method

rmClient It is actually a protocol interface , stay serviceStart Is initialized in

Request to the proxy layer

Through here ClientRMProxy（ all RM The relevant protocol objects will be obtained from the proxy ） Created a protocol Proxy object , It encapsulates the protocol related information and all related RPC operation

This Proxy The object is actually the one mentioned above ApplicationClientProtocolPBClientImpl example , So the call actually goes to ApplicationClientProtocolPBClientImpl#getNewApplication

This proxy The object is really from RPC Class created RPC Agent , Then the request is forwarded through the proxy layer , The request will be given to RM On the deployed machine

RM Accept the request

Now , The request has arrived ClientRMService#getNewApplication

Distribute ID

RM A new ApplicationId. Based on the current cluster timestamp and an incremental Counter As Id

Get resource information

And then through YarnScheduler Get resource information

The request was finally given to ClusterNodeTracker#getMaxAllowedAllocation

You can see , It's direct here clone 了 configuredMaxAllocation Variable , This variable will be YarnScheduler Constantly refreshing .

stay CapacityScheduler#initScheduler in , The maximum resource is set for the first time when initializing the scheduler

stay CapacityScheduler#reinitialize in , When scheduler （ Whether it's fifo still fair） After accepting a new task and allocating resources, the maximum resource parameter will be refreshed

Finally, the result will be returned

client Submit tasks

client Submit the task after completing a series of parameter settings

The logic of request passing is the same as before , So let's talk about the processing logic directly

RM Accept the request

ClientRMService#submitApplication

User permission check

Check whether the application already exists

Prevent duplicate creation of concurrent requests

Check the container Token Is it overdue

Setup queue

from RMAppMaster Accept task processing

Then the task is submitted