Istio best practice: graceful termination

This article excerpts from istio Learning notes

summary

This article is introduced in istio The key points to be paid attention to when implementing elegant termination in the scenario , For some common concerns of container scenarios, please refer to Kubenretes Best practices : End gracefully .

envoy Being forcibly killed leads to abnormal traffic

When the business is istio after , Traffic volume sidecar hijacked , There is no direct connection between processes , But after sidecar This layer of agent :

When Pod Start and stop , It will start from the endpoints Remove... From , No longer forward traffic to it , meanwhile Sidecar I will also receive SIGTERM The signal , Stop accepting at once inbound new connection , But it will keep the stock inbound Connection continues ,outbound Directional flow can still be initiated normally .

But there's one notable detail , if Pod Not quitting soon ,istio The default is to start at the stop 5s After forced killing envoy, When envoy When the process is gone, it cannot forward any traffic ( Whether it's inbound still outbound Direction ), So there may be some problems :

1. If the interface provided by the stopped service takes a long time ( For example, text to speech ), The stock of inbound The request may be disconnected without being processed .
2. If the stopped process needs to call other services ( For example, notify other services to clean up ),outbound The request may fail to invoke .

Customize terminationDrainDuration

istio Provides terminationDrainDuration The custom configuration of the graceful termination time of this connection , Express sleep How long does it take to force the killing envoy, The default is 5s, have access to proxy.istio.io/config This Resource Annotation To configure the service that needs to customize the graceful termination time of the connection terminationDrainDuration, Usage examples :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      annotations:
        proxy.istio.io/config: |
          terminationDrainDuration: 60s #  Custom here  Envoy  Elegant end time 
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 60 #  if  terminationDrainDuration  Overtime  30s  Is specified explicitly  terminationGracePeriodSeconds
      containers:
      - name: nginx
        image: "nginx"

• If terminationDrainDuration Greater than 30s, Need to explicitly Pod Appoint terminationGracePeriodSeconds, Because this value defaults to 30s, namely 30s After that, the process in the container will send a message before exiting SIGKILL The signal will force it to kill . So make sure that terminationGracePeriodSeconds Greater than or equal to terminationDrainDuration Only in this way can the elegant termination duration take full effect .
• terminationDrainDuration The larger the setting is , It also means Pod The slower it will stop , Therefore, it is recommended to customize according to business scenarios , Only reasonably customize the services you need , In other cases, the default values can be used .

Use preStop

If the time required to stop the business is not fixed , Hard to use fixed terminationDrainDuration To control the sidecar Stop time , In fact, you can also give sidecar Add one more preStop Script , In the script, you can indirectly judge whether the application has exited by judging whether it still needs to be connected , After the app exits envoy Just started to quit ( Default wait 5s).

add to preStop It can be modified by sidecar injector Overall situation configmap To achieve :

kubectl -n istio-system edit configmap istio-sidecar-injector

If you use TCM, Managed grid add preStop Background operation of work order is required , The independent grid can modify the configmap, but configmap The name is different from here , Will be suffixed with version .

stay values Inside global.proxy Add the following lifecycle Field :

          "lifecycle": {
            "preStop": {
              "exec": {
                "command": ["/bin/sh", "-c", "while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do sleep 1; done"]
              },
            },
          },

Reference material

• istio common problem : Sidecar Stop the problem
• Kubenretes Best practices : End gracefully