Kubernetes best practice: graceful termination

This article excerpts from kubernetes Learning notes

summary

Pod At the time of destruction , Will stop the process in the container , Usually we need to execute some aftercare logic during the process of stopping , For example, wait for the stock request to be processed to avoid connection interruption , Or notify relevant dependencies to clean up , So as to achieve the goal of elegant termination . This article is introduced in Kubernetes scenario , Best practices for implementing container graceful termination .

Container termination process

Let's first understand the container in Kubernetes Termination process in the environment :

1. Pod Be deleted , The state is set to Terminating.
2. kube-proxy Update forwarding rules , take Pod from service Of endpoint Remove from the list , New traffic is no longer forwarded to the Pod.
3. If Pod Configured with preStop Hook , Will execute .
4. kubelet Yes Pod In each container send out SIGTERM Signal to inform the container process to start and stop .
5. Wait for the container process to stop completely , If in terminationGracePeriodSeconds Inside ( Default 30s) It hasn't completely stopped , Is sent SIGKILL The signal forces the process to kill .
6. All container processes terminate , clear Pod resources .

Business code processing SIGTERM The signal

To achieve graceful termination , Be sure to handle it in the business code SIGTERM The signal , Reference resources Handle SIGTERM Code example .

Do not let shell So that you can't get it SIGTERM The signal

If the container boot entry uses a script ( Such as CMD ["/start.sh"]), The business process becomes shell Can be inherited by child processes. , stay Pod When stopped, the business process may not receive SIGTERM The signal , because shell No automatic signaling to child processes . Please refer to Why can't I get my container SIGTERM The signal ?

If we solve ？ Please refer to Practical skills : stay SHELL Transmit signals in .

The rational use of preStop Hook

If your business code does not handle SIGTERM The signal , Or you can't control the use of third-party libraries or systems to increase the logic of graceful termination , You can also try for Pod Under configuration preStop, The logic of graceful termination is realized here , Example :

        lifecycle:
          preStop:
            exec:
              command:
              - /clean.sh

Reference resources Kubernetes API file

In some extreme cases ,Pod Deleted for a short period of time , New connections may still be forwarded , because kubelet And kube-proxy meanwhile watch To pod Be deleted ,kubelet Are likely to kube-proxy The container has been stopped before the rules are synchronized , This may cause some new connections to be forwarded to the one being deleted Pod, And usually , When the application is SIGTERM No new connections will be accepted , Only keep the stock connection to continue processing , So it may lead to Pod The request to delete the instantaneous part of failed .

In this case , We can also use it preStop First sleep A little , wait for kube-proxy Complete the rule synchronization and then start to stop the processes in the container :

        lifecycle:
          preStop:
            exec:
              command:
              - sleep
              - 5s

Adjust the elegant duration

If the graceful termination time is long (preStop + The business process may stop for more than 30s), It can be customized according to the actual situation terminationGracePeriodSeconds, Avoid being caught early SIGKILL Kill , Example :