Live broadcast review | detailed explanation of koordinator architecture of cloud native hybrid system (complete ppt attached)

author ：

 

Zhang Zuowei   Li Tao

2022  year  4  month , Alibaba cloud native hybrid system  Koordinator  Officially open source . After several months of iteration ,Koordinator  It has been released in succession  4  A version , It can effectively help enterprise customers improve the efficiency of cloud native workload 、 Stability and computational cost .

yesterday （6  month  15  Japan ）, In Alibaba cloud live studio , come from  Koordinator  Zhangzuowei of the community ( Youyi ) 、 Li Tao ( LV Feng ) The two technical experts started from the architecture and characteristics of the project , Shared  Koordinator  How to deal with the challenges in the mixed scenes , In particular, it improves the efficiency and stability of workload operation in the mixed scenario , And the thinking and planning of the subsequent technological evolution . We also sorted out the core content of this live broadcast , I hope to bring you some in-depth inspiration .

Click the link , View live playback now ！

https://yqh.aliyun.com/live/detail/28787

Follow the Alibaba cloud native official account , The background to reply 【0615】 Get complete  PPT

Introduction and development of hybrid technology

The concept of mixed part can be understood from two perspectives , From the node dimension , The mixing part is to deploy multiple containers on the same node , Applications within these containers include both on-line types , Also includes offline types ; From the cluster dimension , The mixed part is to deploy multiple applications in a cluster , Analyze application characteristics through prediction , Realize the peak load shifting and valley filling of business in resource use , To achieve the effect of improving cluster resource utilization .

Based on the above understanding , Then we can define the target problems and technical solutions to be solved by the mixing department . Essentially , Our original intention to implement the mixed department is derived from the unremitting pursuit of resource utilization efficiency in the data center . Accenture report shows ,2011  In, the average machine utilization rate of the public cloud data center was less than  10%, It means that the resource cost of the enterprise is extremely high , On the other hand, with the rapid development of big data technology , Computing jobs have a growing demand for resources . in fact , It has become an inevitable trend for big data to be put into the cloud through the cloud native mode , According to the  Pepperdata  stay  2021  year  12  The survey report of June , A considerable number of enterprise big data platforms have begun to migrate to cloud native technologies . exceed  77%  Of the respondents' feedback is expected to  2021  end of the year , Its  50%  Our big data applications will be migrated to   Kubernetes  platform . therefore , Select batch type task and online service type to apply mixed deployment , Naturally, it has become the general type selection of mixed Department scheme in the industry . Public data display , Through the mixing part , The resource utilization rate of leading enterprises in related technologies has been greatly improved .

Facing the mixed technology , On specific issues of concern , Managers in different roles will have their own priorities .

For administrators of cluster resources , They expect to simplify the management of cluster resources , Realize the resource capacity of various applications , The amount allocated , Clear insight into usage , Improve the utilization of cluster resources , To reduce  IT  The purpose of cost .

For administrators of online applications , They are more concerned about the interference of container hybrid deployment , Because mixed departments are more likely to generate resource competition , The application response time will have a long tail （tail latency）, This leads to the decline of application service quality .

The administrators of offline applications expect that the hybrid system can provide classified and reliable resource oversold , Meet the differentiated resource quality requirements of different job types .

For the above problems ,Koordinator  The following mechanisms are provided , It can fully meet the technical requirements of different roles for the mixed department system ：

• Resource priority and quality of service model for mixed scenarios

• Stable and reliable resource oversold mechanism

• Fine grained container resource orchestration and isolation mechanism

• Scheduling enhancements for multiple types of workloads

• Fast access capability for complex types of workloads

Koordinator  brief introduction

The following figure shows  Koordinator  The overall architecture of the system and the role division of each component , The green part describes  K8s  The components of the native system , The blue part is  Koordinator  The extended implementation based on this . From the perspective of the whole system architecture , We can  Koordinator  It is divided into two dimensions: central control and stand-alone resource management . On the center side ,Koordiantor  Corresponding expansion capability enhancements have been made inside and outside the scheduler ; On the stand-alone side ,Koordinator  Provides  Koordlet  and  Koord Runtime Proxy  Two components , Responsible for the fine management and management of stand-alone resources  QoS  Supportability .

Koordinator  The detailed functions of each component are as follows

• Koord-Manager

• SLO-Controller： Provide resources that are oversold 、 Mixing part  SLO  management 、 Refined scheduling enhances core management and control capabilities .

• Recommender： Provide relevant flexibility for applications around resource portraits .

• Colocation Profile Webhook： simplify  Koordinator  The use of mixed models , The ability to provide one click access for applications , Automatically inject related priority 、QoS  To configure .

• Koord extensions for Scheduler： Enhanced scheduling capability for mixed scenarios .

• Koord descheduler： Provide flexible and extensible rescheduling mechanism .

• Koord Runtime Proxy： As  Kubelet  and  Runtime  Agency between , Meet the resource management needs of different scenarios , Provide a plug-in registration framework , Provide the injection mechanism of relevant resource parameters .

• Koordlet： Responsible for the single machine side  Pod  Of  QoS  guarantee , Provide fine-grained container indicator collection , And interference detection and adjustment strategy , And support a series of  Runtime Proxy  plug-in unit , Isolation parameter injection for refinement .

stay  Koordinator  In the design model of , A core design concept is priority （Priority）,Koordinator  Four levels are defined , Namely  Product、Mid、Batch、Free ,Pod  You need to specify the resource priority of the application , The scheduler will schedule based on the total priority and allocation of each resource . The total resources of each priority will be affected by the resources of high priority  request  and  usage  influence , For example, applied but unused  Product  Resources will be  Batch  Priority reassignment . The specific capacity of each resource priority of the node ,Koordinator  Will be based on the standard  extend-resource  Form update in  Node  In information .

The following figure shows the capacity of each resource priority of a node , The black line  total  Represents the total physical resources of the node , The red line represents the high priority  Product  Actual usage of , The blue broken line to the black straight line reflects  Batch  Priority resource oversold changes , You can see when  Product  When the priority is at the bottom of resource consumption ,Batch  Priority can get more oversold resources . in fact , Aggressive or conservative resource priority policies , Determines the oversold capacity of cluster resources , This point can also be seen from the green line in the figure  Mid  The analysis of resource priority oversold shows that .

As shown in the following table ,Koordinator  With  K8s  The standard  PriorityClass  Form defines the priority of each resource , representative  Pod  The priority of the requested resource . When multi priority resources are oversold , When stand-alone resources are tight , Low priority  Pod  Will be suppressed or expelled . Besides ,Koordinator  It also provides  Pod  Sub priority of level （sub-priority）, It is used for fine control at the scheduler level （ line up , Preemption, etc ）.

Koordinator  Another core concept in the design of is the quality of service （Quality of Service）,Koordinator  take  QoS  Model in  Pod Annotation  Level is defined by extension , It represents the  Pod  Resource quality during stand-alone operation , The main performance is that different isolation parameters are used , When a single machine is short of resources, it will give priority to meeting the high-level requirements  QoS  The needs of . As shown in the following table ,Koordinator  take  QoS  The whole is divided into  System（ System level services ）,Latency Sensitive（ Delay sensitive online services ）,Best Effort（ Resource consuming offline applications ） Three types of , Depending on the sensitivity of the application performance ,Latency Sensitive  Subdivided into  LSE,LSR  and  LS.

stay  Priority  and  QoS  The use of , They are two orthogonal dimensions as a whole , It can be arranged and combined . However, it is affected by the model definition and actual requirements , Some permutations and combinations have constraints . The following table shows some combinations commonly used in mixed scenes , among “O” It means common permutation and combination ,“X” Indicates a permutation that is basically not used .

Examples of actual use of each scenario are as follows .

• Typical scenario ：

• Prod + LS： Typical online applications , It usually requires high application delay , High requirements for resource quality , It is also necessary to ensure a certain resource elasticity .

• Batch + BE： Used for low optimization offline in mixed scenes , Have considerable patience with the quality of resources , For example, batch type  Spark/MR  Mission , as well as  AI  Types of training tasks

• Enhancement of typical scenes ：

• Prod + LSR/LSE： More sensitive online applications , It is acceptable to sacrifice resource elasticity for better certainty （ Such as CPU To nuclear ）, Very high requirements for application delay .

• Mid/Free + BE： And “Batch + BE” The main difference is the different requirements for resource quality .

• Atypical application scenarios ：

• Mid/Batch/Free + LS： For low priority online services 、 Near line calculation and AI Reasoning and other tasks , These tasks are compared with big data type tasks , They cannot accept low resource quality , Interference to other applications is also relatively low ; And compared to the typical online service , They can also tolerate relatively low resource quality , For example, accept a certain degree of expulsion .

Quick Start

Koordinator  Flexible access to mixed units supporting multiple workloads , Here we have  Spark  For example , Introduce how to use the oversold resources of the mixed department . stay  K8s  Running in cluster  Spark  There are two modes of mission ： One is through  Spark Submit  Submit , That is, it is used locally  Spark  Client direct connection  K8s  colony , This method is relatively simple and fast , However, there is a lack of overall management ability , It is often used to develop self-test ; Another way is through  Spark Operator  Submit , As shown in the figure below , It defines the  SparkApplication CRD, be used for  Spark  Job description , The user can go through  kubectl  The client will submit  SparkApplication CR  To  APIServer, Then by  Spark Operator  Responsible for the operation life cycle and  Driver Pod  Management of .

rely on  Koordinator  Capacity enhancement ,ColocationProfile Webhook  Automatically for  Spark  Mission  Pod  Injection related mixing part configuration parameters （ Include QoS,Priority,extened-resource etc. ）, As shown below .Koordlet  Responsible for the single machine side  Spark Pod  The performance of online application will not be affected after mixing , By way of  Spark  Mix with online applications , It can effectively improve the overall resource utilization of the cluster .

# Spark Driver Pod example
apiVersion: v1
kind: Pod
metadata:
 labels:
 koordinator.sh/qosClass: BE
...
spec:
 containers:
 - args:
 - driver
...
resources:
 limits:
 koordinator.sh/batch-cpu: "1000"
 koordinator.sh/batch-memory: 3456Mi
 requests:
 koordinator.sh/batch-cpu: "1000"
 koordinator.sh/batch-memory: 3456Mi
...


Introduction to key technologies

Resource overload  - Resource Overcommitment

In the use of  K8s  When the cluster , It is difficult for users to accurately evaluate the resource usage of online applications , I don't know how to set it better  Pod  Of  Request  and  Limit, Therefore, in order to ensure the stability of online applications , Larger resource specifications will be set . In actual production , The reality of most online applications  CPU  The utilization rate is low most of the time , The high probability is only 10% or 20% , Waste a lot of allocated but unused resources .

Koordinator  Recycle and reuse the allocated but unused resources through the resource overrun mechanism .Koordinator  Evaluate online applications based on metrics data  Pod  How many resources can be recycled （ As shown in the figure above , Marked as  Reclaimed  Part of the is recyclable resources ）, These recyclable resources can be used by low priority workloads , For example, some offline tasks . To make these resources easy for these low priority workloads ,Koordinator  These resources will be updated to  NodeStatus  in （ As shown below  node info）. When an online application has a sudden request to process, it requires more resources ,Koordinator  Through rich  QoS  Enhanced mechanisms help online applications retrieve these resources to ensure quality of service .

# node info
allocatable:
 koordinator.sh/bach-cpu: 50k # milli-core
 koordinator.sh/bach-memory: 50Gi 

# pod info
annotations:
 koordinator.sh/resource-limit: {cpu: “5k”}
resources:
 requests
 koordinator.sh/bach-cpu: 5k # milli-core
 koordinator.sh/bach-memory: 5Gi


Load balancing scheduling  - Load-Aware Scheduling

Over issuance of resources can greatly improve the resource utilization of the cluster , However, it will also highlight the uneven resource utilization among nodes in the cluster . This phenomenon also exists in the non mixed environment , Just because  K8s  The native mechanism does not support resource over issuance , The utilization rate on nodes is often not very high , To some extent, this problem is covered up . But when mixed , This problem is exposed when the resource utilization rate will rise to a relatively high level .

Uneven utilization is generally caused by uneven nodes and local load hotspots , Local load hotspots may affect the overall performance of the workload . The other is on nodes with high load , There may be serious resource conflicts between online applications and offline tasks , Affect the runtime quality of online applications .

To solve this problem , Koordinator  The scheduler provides a configurable scheduling plug-in to control the utilization of the cluster . The scheduling capability mainly depends on  koordlet  Reported node indicator data , Nodes whose load is higher than a certain threshold will be filtered out during scheduling , prevent  Pod  It is impossible to obtain good resource guarantee on this node with high load , On the other hand, it is to avoid the node with high load from deteriorating . Select nodes with lower utilization in the scoring stage . The plug-in will avoid too many problems caused by instant scheduling based on time window and prediction mechanism  Pod  When the cold node machine overheats after a period of time .

Application access management  - ClusterColocationProfile

We are  Koordinator From the beginning of the project, we have considered , Need to reduce  Koordinator  The use threshold of the mixed system , So that we can simply and quickly gray and use the mixed part technology to obtain benefits . therefore  Koordinator  Provides a  ClusterColocationProfile CRD, Through this  CRD  And corresponding  Webhook , You can do this without intruding into components in the inventory cluster , On demand for different  Namespace  Or different workloads , One click to open the mixing ability ,Webhook  Will be based on the  CRD  Describes the rules for newly created  Pod  Automatic injection  Koorinator  priority 、QoS  Configuration and other mixed protocols .

apiVersion: config.koordinator.sh/v1alpha1
kind: ClusterColocationProfile
metadata:
 name: colocation-profile-example
spec:
 namespaceSelector:
 matchLabels:
 koordinator.sh/enable-colocation: "true"
 selector:
 matchLabels:
 sparkoperator.k8s.io/launched-by-spark-operator: "true"
 qosClass: BE
 priorityClassName: koord-batch
 koordinatorPriority: 1000
 schedulerName: koord-scheduler
 labels:
 koordinator.sh/mutated: "true"
 annotations: 
 koordinator.sh/intercepted: "true"
 patch:
 spec:
 terminationGracePeriodSeconds: 30


for instance , It's on it  ClusterColocationProfile  An example of , Means all with  koordinator.sh/enable-colocation=true  Labeled  Namespace  And the  Namespace  Next  SparkOperator  Job created  Pod  Can be transformed into  BE  Type of  Pod（BTW：SparkOperator  Created  Pod  The label will be added  sparkoperator.k8s.io/launched-by-spark-operator=true  Express this  Pod  yes  SparkOperator  conscientious ）.

It is only necessary to follow the following steps to complete the hybrid access ：

$ kubectl apply -f profile.yaml
$ kubectl label ns spark-job -l koordinator.sh/enable-colocation=true
$ # submit Spark Job, the Pods created by SparkOperator are co-located other LS Pods.


QoS  enhance  – CPU Suppress

Koordinator  In order to ensure the runtime quality of online applications in the mixed scenario , On the stand-alone side, a variety of  QoS  Enhance ability .

First of all, I'd like to introduce  CPU Suppress（CPU  Dynamic suppression ） characteristic  . I'd like to introduce it to you , Most of the time, online applications do not completely use up the resources they have applied for , There will be a lot of free resources , These idle resources can be used by newly created offline tasks through resource overload , It can also be used when there are no new offline tasks to be executed on the node , As much as possible to spare  CPU  Resources are shared to off-line tasks in stock . As shown in this figure , When  koordlet  Find that the resources of online applications are idle , And used by offline tasks  CPU  The safety threshold has not been exceeded , Then the idle within the safety threshold  CPU  Can be shared with offline tasks , Make offline tasks faster . Therefore, the load of online applications determines  BE Pod  How much is available in total  CPU. When the online load increases ,koordlet  Will pass  CPU Suppress  suppress  BE Pod, Share  CPU  And online applications .

QoS  enhance  –  Expulsion based on resource satisfaction

CPU Suppress  When the load of online applications increases, offline tasks may be frequently suppressed , This can guarantee the runtime quality of online applications , But it still has some impact on offline tasks . Although offline tasks are of low priority , However, frequent suppression will lead to unsatisfied performance of offline tasks , Seriously, it will also affect the offline service quality . And there are some extreme cases of frequent repression , If the offline task holds special resources such as the kernel global lock when it is suppressed , Such frequent suppression may lead to problems such as priority reversal , Instead, it will affect online applications . Although this does not happen very often .

To solve this problem ,Koordinator  An expulsion mechanism based on resource satisfaction is proposed . We put the actual distribution of CPU Total amount   And   Expected distribution of  CPU  The ratio of the total amount becomes  CPU  Satisfaction . When offline task group  CPU  Satisfaction is below the threshold , And the offline task group  CPU  Utilization over  90%  when ,koordlet  Will expel some low priority offline tasks , Free up some resources for higher priority offline tasks . This mechanism can improve the resource requirements of offline tasks .

QoS  enhance  - CPU Burst

We know  CPU  Utilization is a period of time  CPU  Average value used . And most of the time we observe statistics in a coarse time unit granularity  CPU  utilization , It was observed at this time  CPU  The change of utilization rate is basically stable . But if we observe statistics in a finer time unit granularity  CPU  utilization , You can see  CPU  The burst characteristics used are very obvious , It's not stable . As shown below  1s  Granularity observation utilization （ violet ） and  100ms  Utilization of granularity observation （ green ） contrast .

The observation of fine-grained data shows that CPU  Outburst and repression are the norm .Linux Through the kernel  CFS  Bandwidth controller  cgroup CPU  Consumption of , It limits  cgroup  Of  CPU  Consumption cap , Therefore, it is often encountered that some services are severely damaged in a short time under sudden traffic  throttle, Long tail delay , The service quality is reduced , As shown in the figure below ,Req2  because  CPU  Suppressed , Postpone to the  200ms  To be dealt with .

To solve this problem ,Koordinator  be based on  CPU Burst  Technology helps online applications cope with emergencies .CPU Burst  Allow workload to be used when there is burst request processing CPU  Resource time , Use everyday  CPU  resources . For example, the container is used in daily operation  CPU  Resource does not exceed  CPU  Current limiting , spare CPU Resources will be accumulated . Later, when the container runs, it needs a lot of  CPU  Resource time , Will pass through  CPU Burst  The function is used suddenly  CPU  resources , The resources used suddenly come from the accumulated resources . As shown in the figure below , Suddenly  Req2  Because there is accumulated  CPU  resources , adopt  CPU Burst  The function can avoid being  throttle, Processed the request quickly .

QoS  enhance  – Group Identity

In the mixed scenes ,Linux  Although the kernel provides a variety of mechanisms to meet the scheduling requirements of workloads with different priorities , But when an online application and an offline task are running on a physical core at the same time , Because offline tasks share the same physical resources , The performance of online applications will inevitably be disturbed by offline tasks, resulting in performance degradation .Alibaba Cloud Linux 2  From kernel version  kernel-4.19.91-24.al7  Start supporting  Group Identity  function ,Group Identity  Is a kind of  cgroup  Group as a unit to achieve the means of scheduling special priority , In short , When online applications need more resources , adopt  Group Identity  It can temporarily suppress offline tasks to ensure that online applications can respond quickly .

It's easy to use this feature , You can configure the  cpu cgroup   Of  cpu.bvt_warp_ns  that will do . stay  Koordinator  in ,BE  The corresponding configuration of class offline task is  -1, That is, the lowest priority , LS/LSR  And other online application types are set to  2, That is, the highest priority .

QoS  enhance  – Memory QoS

Containers have the following two constraints when using memory ：

• Own memory limit ： When the container's own memory （ contain Page Cache） Close to the upper limit of the container , Will trigger the kernel's memory reclamation subsystem , This process will affect the performance of memory application and release in the container .

• Node memory limit ： When container memory is oversold （Memory Limit>Request） The memory of the whole machine is insufficient , Will trigger the global memory reclamation of the kernel , This process has a great impact on performance , Extreme conditions even lead to abnormal operation of the whole machine .

To improve application runtime performance and node stability ,Koordinator  introduce Memory QoS  Ability , Improve memory performance for applications . When the function is turned on ,koordlet  According to the adaptive configuration memory subsystem （Memcg）, On the basis of ensuring the fairness of node memory resources , Optimize the performance of memory sensitive applications .

Follow up evolution plan

Refine  CPU  layout  - Find-grained CPUOrchestration

We are designing and implementing refinement  CPU  The choreography mechanism .

Why should we provide this orchestration mechanism ？ With the improvement of resource utilization, it enters the deep-water area of the mixed department , The performance of the resource runtime needs to be further tuned , More sophisticated resource orchestration can better ensure the quality of runtime , Thus, the utilization ratio will be pushed to a higher level through mixing .

We put  Koordinator QoS  Online applications  LS  Types are divided in more detail , It is divided into  LSE、LSR  and  LS  Three types of . After the split  QoS  Types have higher isolation and runtime quality . Through such a split , Whole  Koordinator QoS  More precise and complete semantics , And compatible  K8s  Existing  QoS  semantics .

And we aim at  Koordinator QoS, Designed a rich and flexible  CPU  Choreography strategy , As shown in the following table .

Koordinator QoS  Corresponding  CPU  Choreography strategy

in addition , in the light of  LSR  type , There are also two binding strategies , It can help users balance performance and economic benefits .

• SameCore  Strategy ： Better isolation , But the elastic space is small .

• Spread  Strategy ： Moderate isolation , But it can be optimized by other isolation strategies ; If you use it properly, you can get a better result than  SameCore  Strategy for better performance ; There is a certain elastic space .

Koordinator  This set of refinement  CPU  The orchestration is compatible  K8s  Existing  CPUManager  and  NUMA Topology Manager  The mechanism . That is to say, the stock cluster uses  Koordinator  Will not affect the stock  Pod, It can be safely used in grayscale .

Reserve resources  - Resource Reservation

Resource reservation is another feature we are designing . Resource reservation can help solve the pain points of resource management . For example, sometimes it looks like the familiar Internet business scenarios , Have very strong peak valley characteristics . Then we can reserve resources before the peak to ensure that there must be resources to meet the peak request . In addition, you may also encounter problems during capacity expansion , After the capacity expansion is initiated, there are no resources  Pod  Just  Pending  In the cluster , If you can confirm whether resources are available before capacity expansion , Adding a new machine when there are no resources will give you a better experience . There are also rescheduling scenarios , The expelled can be guaranteed through resource reservation  Pod  There must be resources available , It can greatly reduce the resource risk of rescheduling , More secure use of rescheduling capabilities .

Koordinator  The resource reservation mechanism of does not intrude  K8s  What the community already has  API  And code . And support  PodTemplateSpec, Imitate a Pod  Find the most suitable node through the scheduler . And support the way of declaring ownership  Pod  Give priority to reserved resources , For example, when a real  Pod  When scheduling , Will give priority to trying according to  Pod  Find the appropriate reserved resources , Otherwise, continue to use the idle resources in the cluster .

Here's a  Reservation CRD  Example （ With a final  Koordinator  The design adopted by the community shall prevail ）

kind: Reservation
metadata:
 name: my-reservation
 namespace: default
spec:
 template: ... # a copy of the Pod's spec
 resourceOwners:
 controller:
 apiVersion: apps/v1
 kind: Deployment
 name: deployment-5b8df84dd
 timeToLiveInSeconds: 300 # 300 seconds
 nodeName: node-1
status:
 phase: Available
 ...


Refine  GPU  Dispatch  - GPU Scheduling

Refine  GPU  Scheduling is a capability we expect to provide in the future .GPU  and  CPU  There are great differences in resource characteristics , And in model training scenarios like machine learning , A training task will have different performance differences due to different topologies , For example, according to the machine learning task  worker  Different topological combinations between , Will get different performance , This is not only reflected between nodes in the cluster , And even on a single node ,GPU  Between cards also because of whether to use  NVLINK  There will also be huge performance differences , This makes the whole  GPU  The scheduling and allocation logic of becomes very complex . and  GPU  and  CPU  When the computing task of is mixed in the cluster , How to avoid the waste of two kinds of resources , It is also an optimization problem that needs to be considered .

Recommended specifications  - Resource Recommendation

follow-up  Koordinator  It also provides the ability to recommend specifications based on portraits . As mentioned earlier , It is difficult for users to accurately evaluate the resource usage of an application ,Request  and  Limit  What is the relationship , How to set it  Request/Limit, Which combination is the most appropriate for my application ？ Often overestimate or underestimate  Pod  Resource specifications , Lead to resource waste and even stability risk .

Koordinator  Will provide resource portrait capability , Collect and process historical data , Recommend more accurate resource specifications .

Community building

So far, , We have released four versions in the last two months . The previous versions mainly provide resource overload 、QoS  Enhanced ability , And open source new components  koord-runtime-proxy. stay  0.4  In the version , We started working on the scheduler , First, the load balancing scheduling capability is opened . at present  Koordinator  The community is realizing  0.5  edition , In this version ,Koordinator  Will provide refinement  CPU  The ability to orchestrate and reserve resources , In the future planning , We will be rescheduling 、Gang  Dispatch 、GPU  Dispatch 、 elastic  Quota  And so on .

We are looking forward to your use  Koordinator  Positive feedback on any problems encountered 、 Help improve documentation 、 Repair  BUG  And add new features

• If you find a typo, try to fix it!

• If you find a bug, try to fix it!

• If you find some redundant codes, try to remove them!

• If you find some test cases missing, try to add them!

• If you could enhance a feature, please DO NOT hesitate!

• If you find code implicit, try to add comments to make it clear!

• If you find code ugly, try to refactor that!

• If you can help to improve documents, it could not be better!

• If you find document incorrect, just do it and fix that!

• ...

Besides , We also on Tuesday  19:30  to  20:30  Organized regular biweekly community meetings , Welcome like-minded partners to add an exchange group to learn more .

Wechat group

Nail group

Click on

here

, Learn now  Koordinator  project ！