CODING CD

Preface

This article will show you how to pass CODING CD Use Nginx Ingress To achieve Blue green release .

Why blue and green publishing ？ With the rapid development of business , The demand for the development team is getting higher and higher , On the one hand, we need to provide users with stable services , On the one hand, fast business iteration is required . Therefore, based on the comprehensive consideration of system stability and fast business iteration , We need to use the blue-green way to release the new version of the service , Realize the smooth upgrade of application services .

Why use CODING CD？ The traditional deployment is to modify YAML The mirror version of the file , And then through the command line kubectl apply

Update the version of the application service in the same way , This way of publishing relies too much on manual execution , about DevOps It's intolerable for the team . And by CODING CD

Deploy process to realize automatic pipeline , All stages of the assembly line can be checked by anyone in the team 、 Improvement and validation , The development team can improve the speed of release and reduce the risk and cost of release .

summary

What is? Nginx Ingress

Nginx Ingress yes Kubernetes Ingress An implementation of , It passes through watch Kubernetes Clustered Ingress resources , take

Ingress The rules are translated into Nginx Configuration of , And then let Nginx To carry out 7 Layer traffic forwarding .

Use annotations to explain

We pass to Ingress Resource assignment Nginx Ingress Some of the annotation It can be released in blue and green , You need to create two

Ingress, A normal Ingress（myapp-ingress）, The other is to take

nginx.ingress.kubernetes.io/canary: "true" This is fixed annotation Of

Ingress, Let's call it Canary Ingress（myapp-blue-ingress）, Generally represents a new version of the service , Combined with other traffic segmentation strategy

annotation The blue and green publishing of multiple scenarios can be realized by configuring together , The following is the application of this practice annotation Introduction to ：

• nginx.ingress.kubernetes.io/canary-by-header: If the request header contains the header name , And the value of always Words , Forward the request to the Ingress Corresponding back-end services defined ; If the value is never No forwarding , Can be used to roll back to the old version ; If it is any other value, the annotation.
• nginx.ingress.kubernetes.io/canary-by-header-value: This can be used as an example canary-by-header A supplement to , The value of request header can be customized to other values , It's no longer just always or never; When the value of the request header hits the custom value here , The request will be forwarded to the Ingress Corresponding back-end services defined , If it is any other value, the annotation.
• nginx.ingress.kubernetes.io/canary-weight: Express Canary Ingress Percentage of the proportion of traffic allocated , Value range 0-100, For example, set to 10, It means distribution 10% Flow rate of Canary Ingress Corresponding back-end services .

More about Nginx Ingress

Please refer to the official documents for the notes of (https://links.jianshu.com/go?to=https%3A%2F%2Fkubernetes.github.io%2Fingress-

nginx%2Fuser-guide%2Fnginx-configuration%2Fannotations%2F%23canary).

What is a blue-green release

Blue green release , It's outside the stable cluster of production environment , Deploy an additional new cluster of the same size as the stable cluster , And through flow control , Gradually introduce traffic to the new cluster until

100%, The original stable cluster will remain online for a period of time at the same time as the new cluster , Any exception occurred during this period , All traffic can be switched back to the original stable cluster immediately , Fast implementation of rollback . Until all verification is successful , The old stable cluster , The new cluster becomes the new stable cluster .

Publishing process

image

Blue and green release process , Include ： The blue and green release begins 、 Blue and green 、 Blue green verification 、 Blue and green cancel or go online .

image

As shown in the figure above , The version of the old cluster is v1, The cluster uses version=v1 The label is Service myapp-svc Access to the .

• An additional new cluster has been deployed , Version is v2, The cluster uses version=v2 The label is Service myapp-blue-svc Access to the .
• Through flow control , Control some or all traffic to the new cluster , Blue green validation , At the same time, the original stable cluster remains online .
• If the verification does not find any faults , The application of myapp The blue and green release of ,v2 Version cluster becomes a new stable cluster .
• If a fault is found in the verification , Through flow control , Switch all traffic back to the original stable old cluster , Fast implementation of rollback .

practice

Prerequisite ：

Kubernetes It needs to be deployed in the cluster Nginx Ingress As Ingress Controller, And exposed the unified flow entrance to the outside .

Old cluster initialization

Publishing process

image

To configure

image

Here is the configuration deployment myapp What application services need docker Mirror products , There are also startup parameters version, This parameter is mainly used as a label for Service

Provide filtering conditions , Realize the flow control of new and old version services .

Release strategy

image

The publishing strategy adopts Manual confirmation Stage , Configurable confirmer , Choose different publishing methods .

General release

image

Regular release uses Preconditions check Stage , Preset configuration #judgment(" Release strategy ") == ' General release ' expression , In the expression [judgment

function ](https://links.jianshu.com/go?to=https%3A%2F%2Fspinnaker.io%2Freference%2Fpipeline%2Fexpressions%2F%23judgmentstring) return Manual confirmation The options selected in the phase , And pass

== Relational operators to compare values in expressions , If the expression is successful , The phase that will execute this branch .

New clusters ( Green cluster ) Deploy

image.png

${parameters.version} The expression represents the configured startup parameter in the reference configuration version.

This is where the new cluster is deployed Deployment Be hit with Spinnaker annotation ：

• strategy.spinnaker.io/versioned, When the annotation is set to ‘true’ when ,Spinnaker Will be right Deployment Version management of resources . Simply speaking , It's after the release Deployment The name will have a version number , Such as myapp-deploy-v000.
• strategy.spinnaker.io/max-version-history, The note indicates Spinnaker The number of resource versions to keep will be configured . When Spinnaker released k8s The number of historical versions of the resource exceeds max-version-history when ,Spinnaker Will delete the old version of the resource . I'm going to set it to '3', I want to keep it 3 It's a historical version of Deployment myapp-deploy.

Be careful , This stage also requires binding the configured myapp-image Docker Mirror products ,Spinnaker Will be replaced automatically Manifest Match in

image. Refer to the help document for specific binding replacement rules (https://links.jianshu.com/go?to=https%3A%2F%2Fhelp.coding.net%2Fdocs%2Fcd%2Fpipe%2Fartifacts%2Fin-

kubernetes.html%23%25E5%259C%25A8-manifest-%25E4%25B8%25AD%25E7%25BB%2591%25E5%25AE%259A%25E5%2588%25B6%25E5%2593%2581).

Switching traffic

image.png

here Service myapp-svc adopt version: '${parameters.version}' Tag access to... With the same tag myapp-

deploy Pod,myapp-ingress Configure routing to Service myapp-svc The rules of , It can be done by

http://myapp.coding.prod

Access to the myapp-svc The back-end service .

Old cluster offline

image

Here the Expansion and contraction capacity (Manifest) Stage , choice Dynamic selection of targets also Target choice Next new , take Deployment myapp-deploy

Of replicas Set to 0, Complete the offline operation of the old cluster . Why not use Delete (Manifest) The stage is to keep myapp-deploy

Historical release version of , Easy to roll back , Delete... In addition myapp-deploy The historical version of the `strategy.spinnaker.io/max-version-

history` Annotation implementation .

Be careful , here Execution Options Of If the phase fails Option selection Terminate this branch of the process , Because for the old cluster initialization deployment , There is no new version available for offline operation , This phase will fail , Cause the whole process deployment to fail .

Why is there more of this phase in regular releases ？ The goal is to make the pipeline reusable , When the new version service is released for the second time, it can also be implemented through the deployment process , The new cluster is deployed until the service is stable , It can be done at the same time Switching traffic and Old cluster offline operation .

The blue and green release begins

image

stay “ Release strategy ” Select the release type in the manual confirmation phase of “ Blue green release ”, You can start the blue and green release . stay “ Blue green release ” Configure condition expressions in the precondition check phase of ：`#judgment(" Release strategy ")

== ' Blue green release '`.

Blue and green

New clusters ( The blue cluster ) Deploy

image.png

Blue and green technological process ： First push the routing rules , Control all traffic in the old cluster , Ensure that no traffic will be received during the startup of the new cluster deployment .

We use examples version: '${parameters.version}'

Tag to identify whether it's a blue cluster or a green cluster , Therefore, you need to fill in different startup parameters before starting the deployment process , Make sure Service myapp-blue-svc Through different version

The tag value is filtered to the instance of the blue cluster Pod; in addition , By publishing Ingress The rules myapp-blue-ingress, Request header satisfied

blueGreenVersion=blue Matching requests will be routed to the blue cluster Service myapp-blue-svc. After pushing the rules , You can deploy the new cluster .

Be careful , This phase also needs to bind the configured myapp-image Docker Mirror products .

Users need to match the request header correctly before they can access the new version service ：`curl -H 'blueGreenVersion:blue'

http://myapp.coding.prod`.

Blue green verification

Blue green verification technological process ： Once the initialization is complete , You can push routing rules , Route part or all of the request traffic to the new cluster for verification . If there are no problems in the verification process , Traffic can be continuously migrated to new clusters , Until all traffic is in the new cluster . If the blue-green verification is successful, enter Blue and green complete the launch technological process , If the verification fails, enter Blue and green cancel technological process .

Blue green verification

image

Blue green validation uses Manual confirmation Stage , Configuration confirmer , You can choose to control part of the request traffic or route all the traffic to the new cluster for verification . Custom parameters are also configured here

blue_ratio, This parameter is the percentage of requests that control part of the traffic to the new cluster .

Control part of the traffic to the new cluster

image.png

#stage(" Blue green verification ")["context"]["customParams"]["blue_ratio"]

Expressions represent references Manual confirmation Custom parameters configured in phase blue_ratio, This will control random blue_ratio% Request to enter the new version cluster .

At this stage Execution Options You also need to configure conditional expressions #judgment(" Blue green verification ") == ' Control part of the traffic to the new cluster ', Control the direction of branch execution of the deployment process .

image

Blue and green validation results confirm

c959fI.png

If problems are found in the process of controlling part of the traffic to the new cluster verification , All traffic can be switched back to the old cluster through this manual confirmation phase .

At this stage Execution Options You also need to configure conditional expressions `#judgment(" Blue green verification ") ==

' Control part of the traffic to the new cluster '`, Because the previous phase was skipped because the conditions were not met , This stage is the downstream stage of the previous stage , Will continue to carry out , So you also need to configure conditional expressions .

Control all traffic to the new cluster

image.png

explain ： By setting nginx.ingress.kubernetes.io/canary-weight: '100' Control all traffic to the new cluster .

Be careful , At this stage Execution Options You need to configure conditional expressions `${ (#stage(" Blue green verification ")"status".toString() ==

"SUCCEEDED" && #judgment( ' Blue green verification ') == ' Control all traffic to the new cluster ') ||

(#stage(" Blue and green validation results confirm ")"status".toString() == "SUCCEEDED" &&

#judgment(' Blue and green validation results confirm ') == ' Verify success , Control all traffic to the new cluster ') }`. In the expression [stage

function ](https://links.jianshu.com/go?to=https%3A%2F%2Fspinnaker.io%2Freference%2Fpipeline%2Fexpressions%2F%23stagestring)

Use #stage(" Blue green verification ")["status"].toString() return " Blue green verification " The execution status of the manual confirmation phase .Spinnaker

The syntax of expressions is based on [Spring Expression Language

(SpEL)](https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.spring.io%2Fspring%2Fdocs%2Fcurrent%2Fspring-

framework-reference%2Fhtml%2Fexpressions.html), have access to && and || Equirelational operators .

We need such a complex conditional expression here , It's because you can start from “ Blue green verification ” The manual confirmation phase directly selects Control all traffic to the new cluster Option to enter this stage , You can also get it from “ Blue and green validation results confirm “ Manual confirmation phase selection Verify success , Control all traffic to the new cluster Option to enter this stage , Therefore, powerful conditional expressions are needed to control the direction of process branches . thus it can be seen ,Spinnaker

The power of the deployment process , It can support complex process branch control , Meet the different needs of various release scenarios . And the pipeline only needs to be configured once , Multiple execution .

The final effect of blue and green is confirmed

c95F6f.png

If problems are found in the process of controlling all traffic to the new cluster verification , You can enter through this manual confirmation phase “ Blue and green cancel ” technological process , Switch all traffic back to the old cluster .

At this stage Execution Options It also needs to be configured with “ Control all traffic to the new cluster ” Stage like conditional expression , To control the direction of branch execution of the deployment process .

Blue and green complete the launch

The new cluster is verified successfully

c95ptA.png

The new cluster verification successfully adopted Preconditions check Stage , The configuration of preset conditions is relatively simple ：#judgment(" The final effect of blue and green is confirmed ") == ' The new cluster is verified successfully '.

Blue and green complete the launch

image.png

If the blue-green verification does not find a problem , Then you can complete the blue and green release online . Only when all the traffic is in the new cluster can the operation be completed . After the launch , If you need to roll back again , That's the normal rollback process .

here Service myapp-svc adopt version: '${parameters.version}' Tag access to a new cluster with the same tag

Pod, Achieve traffic switching ,myapp-blue-ingress Also configure routing to Service myapp-svc The rules of .

Old cluster offline

c95Ppt.png

“ Old cluster offline ” With the regular release of “ Old cluster offline ” be similar , All use Expansion and contraction capacity (Manifest) Stage , It's just Execution Options Use the default configuration .

Blue and green cancel

New cluster validation failed

c95i1P.png

New cluster validation failed Preconditions check Stage , The preconditions are complex ：`${ (#stage(" Blue and green validation results confirm ")"status".toString() ==

"SUCCEEDED" && #judgment( ' Blue and green validation results confirm ') == ' Validation failed , Blue and green cancel ') ||

(#stage(" The final effect of blue and green is confirmed ")"status".toString() == "SUCCEEDED" &&

#judgment(' The final effect of blue and green is confirmed ') == ' New cluster validation failed ')

}. Because you can start from “ Blue and green validation results confirm ” The manual confirmation phase directly selects Validation failed , Blue and green cancel Option to enter this stage , You can also get it from “ Blue green final effect confirmation manual confirmation stage selection New cluster validation failed ` Option to enter this stage .

Blue and green cancel

image.png

If problems are found in the blue-green validation process , The blue-green cancellation can switch all traffic back to the old cluster in seconds , Specific steps ： Push routing rules control all traffic to the old cluster .

Modify here myapp-blue-ingress Routing rules , Only the request header matches blueGreenVersion=blue

To access the new cluster , If it doesn't match, visit the old cluster , All traffic is switched back to the old cluster .

New cluster offline

c95EnS.png

Here the Delete (Manifest) Stage , choice Dynamic selection of targets also Target choice Abreast of the times , Put the new cluster's Deployment myapp-

deploy Delete , Complete the offline operation of the old cluster .

Of course , You can also modify the routing rules first , First, switch the traffic back to the old cluster , Then keep the scene , Facilitate troubleshooting .

effect

Finally, the deployment process of blue and green release is configured , Next, let's look at the blue and green release effect .

Initialize the green cluster

17(1).gif

The blue and green release was successful

18.gif

Blue green release failed

19.gif

Conclusion

In the example above , We passed the CODING CD Configure a pipeline in , Based on Nginx Ingress

Blue and green release of . The deployment process only needs to be configured once , You can use it forever , It greatly reduces the mistakes caused by manual deployment , Improve the release efficiency of the application .