New progress in the construction of meituan's Flink based real-time warehouse platform

The main contents include ：

1. Status quo of platform construction

2. Problems encountered and solutions

3. The future planning

01 Status quo of platform construction

Meituanyu 2018 First introduced in Flink Real time computing engine , At that time, the concept of real-time data warehouse was not very popular , The platform only provides Flink Jar Task lifecycle management and monitoring alarms .

2019 year , We have noticed that the main application scenario of real-time computing is to solve the problem of low timeliness of offline data warehouse . Offline data warehouse is mature , adopt SQL Method development is very simple , The real-time part of the data warehouse is mainly through Flink DataStream API To develop , The threshold is quite high , And compared with the development mode of offline data warehouse, it is fragmented . therefore , We began to investigate the solution of real-time data warehouse , The goal is to lower the development threshold , And try to promote FlinkSQL, Finally, the real-time data warehouse platform of meituan was named NAU.

2020 year , The meituan real-time data warehouse platform was officially launched . It provides... To the business FlinkSQL Job development portal , Mainly responsible for two aspects of work ：

• First , Align the common data sources of real-time data warehouse with the concept of offline table , Manage with a data model ;

• secondly , Provide FlinkSQL Develop supporting efficiency tools , For example, check and debug functions .

But in the actual promotion process , We found the business in FlinkSQL The threshold for operation and maintenance is still relatively high , therefore , We will focus on the next work