Istio FAQ: sidecar stop sequence

This article excerpts from istio Learning notes

background

Istio stay 1.1 There was a problem before version : Pod At the time of destruction , If the process continues to call other services during the exit process ( For example, notify another service to clean up ), Fail to call .

For more details, please refer to issue #7136: Envoy shutting down before the thing it's wrapping can cause failed requests .

reason

Kubernetes In the process of destruction Pod In the process of , Will send to all containers at the same time SIGTERM The signal , therefore Envoy Start and stop at the same time as the business container ,Envoy No new traffic will be accepted during the stop process , And because of Istio Traffic hijacking , All outgoing flows will pass through Envoy Forward , If Envoy Do not accept new traffic , It will cause the business to call other services to fail .

Community solutions

If Kubernetes Self support container dependency management , Then this problem can be solved naturally . The community also proposed Sidecar Container Characteristics of , Unfortunately, it was finally abandoned , The new plan has not yet been implemented , Details available This note .

Later, along with istio Community promotion , Some optimizations have been made for elegant termination scenarios :

• 2019-02: Liam White Submit PR Envoy Graceful Shutdown , Give Way Pod In the process of stopping Envoy Can achieve graceful stop ( Keep the stock connection and continue processing , But reject all new connections ), wait for terminationDrainDuration Stop after a long time envoy example . The PR Finally it is merged into istio 1.1.
• 2019-11: Rama Chavali Submit PR move to drain listeners admin endpoint , take Envoy The graceful stop method is changed from hot restart to calling Envoy Later, it provided by itself admin Interface (/drain_listeners?inboundonly) , The point is to bring inboundonly Parameters , That is, just refuse inbound New connection of direction ,outbound The new connection can still be initiated normally , It also makes Pod During the stopping process, the business process continues to call other services to realize . The PR Finally it is merged into istio 1.5.

So in istio 1.5 And above , stay Pod A short period of time during a stop ( Default 5s), Business processes can still make requests to other services .

Best practices

Customize elegant duration

If your business needs to call other services during the stop process , Use istio 1.5 There is usually no problem with the above versions without any additional configuration , Because it will default to 5s Elegant end time , This duration is sufficient for most scenarios . If the business is special , It may take a long time to stop ( exceed 5s), And you need to initiate calls to other services , In this case, it is recommended to use istio 1.7 And above , Support use proxy.istio.io/config This Resource Annotation To configure the service that needs to customize the graceful termination time terminationDrainDuration, Usage examples :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      annotations:
        proxy.istio.io/config: |
          terminationDrainDuration: 60s #  Custom here  Envoy  Elegant end time 
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 60 #  if  terminationDrainDuration  Overtime  30s  Is specified explicitly  terminationGracePeriodSeconds
      containers:
      - name: nginx
        image: "nginx"

It should be noted that , If terminationDrainDuration Greater than 30s, Need to explicitly Pod Appoint terminationGracePeriodSeconds, Because this value defaults to 30s, namely 30s After that, the process in the container will send a message before exiting SIGKILL The signal will force it to kill . So make sure that terminationGracePeriodSeconds Greater than or equal to terminationDrainDuration Only in this way can the elegant termination duration take full effect .

Use preStop To avoid

If the time required to stop the business is not fixed , It is not easy to use a fixed elegant duration , You can also give sidecar Add one more preStop Script , In the script, you can indirectly judge whether the application has exited by judging whether it still needs to be connected , After the app exits envoy Just to really quit .

add to preStop It can be modified by sidecar injector Overall situation configmap To achieve :

kubectl -n istio-system edit configmap istio-sidecar-injector

If you use TCM ( Tencent cloud service grid ), Managed grid add preStop Background operation of work order is required , The independent grid can modify the configmap, but configmap The name is different from here , Will be suffixed with version .

stay values Inside global.proxy Add the following lifecycle Field :

          "lifecycle": {
            "preStop": {
              "exec": {
                "command": ["/bin/sh", "-c", "while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do sleep 1; done"]
              },
            },
          },