Didi elasticsearch cluster cross version upgrade and platform reconfiguration

Reading guide ： Not long ago , sound of dripping water ES The team will maintain 30 Multiple ES colony ,3500 Multiple ES node ,8PB The data of , from 2.3.3 Seamless upgrade to 6.6.1. Under the premise of basically zero impact and change on user query writing , It's solved ES Cross major protocol incompatibility 、 Incompatible file format 、mapping Incompatibility, etc , The whole process is completely transparent to most users . At the same time, it's done Arius Architecture upgrade , The single machine query performance has been improved 40%, The whole cluster cpu falling 10%, write in tps promote 30%, The utilization rate of cluster resources has been improved 20%、0 fault 、 The cost of operation and maintenance has decreased 60% The achievement of .

This paper will systematically introduce didi from 2.3.3 Upgrade to across large versions 6.6.1 Problems encountered in the process and solutions , And in the search platform construction process of systematic thinking .

01 Background introduction

1. The cluster size

At present, Didi uses ES The version is 2.3.3, The number of clusters is 40 Multiple , Node scale has 3500+, The total capacity of the cluster is 8PB.

2. Business scale

1200 Multiple platform applications are using ES,30 Multiple core applications are using ES, Written in TPS Yes 1500W, Of the query QPS Yes 25W.

02 Problem analysis

For the above scale ES colony , from 2.3.3 Upgrade to 6.X edition , The small version will be determined based on the results of the final analysis , Potential problems need to be analyzed and distinguished .

1. Problem analysis

First of all, it analyzes the four problem areas ：

• Engine side ： As a result of 2.3.3 Upgrade to 6.X edition , The version gap is too big , Not compatible in file format or protocol , So you can't roll in place and upgrade directly , Need to double write relocation and upgrade , It costs a lot of machines to get involved
• The user side ：6.X The version gradually does not support TCP Interface , Therefore, users need to adapt and upgrade ; There are also some differences between query and return value , If the user side adapts , Will greatly affect the progress of the upgrade
• Resource side ： Because you can't scroll directly in place, upgrade directly , Need to double write using a lot of machines , But we can't provide the machines needed to upgrade , If resources cannot be guaranteed during the upgrade process , That will also greatly affect the progress of the upgrade
• Operation side ： How to control the operation and maintenance of the new version of multi cluster ？ How to verify the results of the upgrade ？ How to guarantee the efficiency and quality of query ？ All these problems need to be considered

Link to the original text :【https://www.infoq.cn/article/J6BFzWfJA0cbmgzN7R1w】. Without the permission of the author , Prohibited reproduced .