Istio FAQ: sidecar startup sequence

This article excerpts from istio Learning notes

background

Some services are going istio In the process of migration and transition , Sometimes you may meet Pod Boot failure , Then keep restarting , The reason for troubleshooting is that other services need to be called during business startup ( For example, pull the configuration from the configuration center ), If you fail, quit , There is no retry logic . The reason why the call failed is envoy Not ready yet (envoy It is also necessary to pull the configuration from the control surface , It will take a little time ), The traffic sent by the service cannot be processed , So the call fails ( Reference resources k8s issue #65502 ).

Best practices

The current best practice for this kind of problem is to make the application more robust , Add retry logic , Don't quit immediately after the call fails , If it is troublesome to change , You can also add... Before starting the command sleep, Wait a few seconds ( It may not be very elegant ).

If you don't want to make any changes to the application , You can also refer to the following evasion scheme .

Avoid scheme : adjustment sidecar Injection sequence

stay istio 1.7, Community through to istio-injector The injection logic adds a called HoldApplicationUntilProxyStarts To solve this problem , When the switch is on ,proxy Will be injected into the first container.

sidecar-order-1.png

sidecar-order-2.png

see istio-injector Automatic injection uses template, You can know that if you open HoldApplicationUntilProxyStarts Will be for sidecar Add one postStart hook:

2e3bb746.png

Its purpose is to block the subsequent business container startup , Wait until the sidecar Start the business container after it is fully started .

This switch configuration is divided into global and local , Here are the enabling methods .

Global configuration :

modify istio Of configmap Global configuration :

kubectl -n istio-system edit cm istio

stay defaultConfig Lower join holdApplicationUntilProxyStarts: true

apiVersion: v1
data:
  mesh: |-
    defaultConfig:
      holdApplicationUntilProxyStarts: true
  meshNetworks: 'networks: {}'
kind: ConfigMap

If you use IstioOperator,defaultConfig modify CR Field meshConfig:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  meshConfig:
    defaultConfig:
      holdApplicationUntilProxyStarts: true

Local configuration :

If you use istio 1.8 And above , It can be used for Pod add proxy.istio.io/config annotation , take holdApplicationUntilProxyStarts Set as true, Example :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      annotations:
        proxy.istio.io/config: |
          holdApplicationUntilProxyStarts: true
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: "nginx"

It should be noted that , When you turn on this switch , This means that the business container needs to wait sidecar Completely ready Before starting , Will make Pod Start slower . It may be difficult to cope with sudden traffic scenarios when rapid capacity expansion is required , So the suggestion is to evaluate the business scenario by yourself , Using the method of local configuration , Only turn on this switch for the business you need .

Perfect plan : K8S Support container dependency

The most perfect solution is Kubernetes Self support container dependencies , The community also proposed Sidecar Container Characteristics of , Unfortunately, it was finally abandoned , The new plan has not yet been implemented , Details available This note .

Reference material

• Istio Operation and maintenance practical series （1）： Apply container to Envoy Sidecar The startup dependency of
• PR: Allow users to delay application start until proxy is ready
• Kubernetes Sidecar Containers Feature survey notes