Modern data architecture selection: Data fabric, data mesh

author ｜QCon

Data architecture is always on the way of updating iteration , So that it can quickly adapt to the changing data environment , More agile and large-scale delivery of data to business units . In traditional data architecture , There is high data complexity 、 Lack of agility 、 Not convenient for collaboration 、 Data consistency and low interpretability . These challenges hinder enterprises from moving towards data-driven enterprises , It is also difficult to achieve rapid response to business needs .

In the process of seeking the best data architecture ,Data Fabric and Data Mesh Often noticed , At first glance, the two are very similar , But there are fundamental differences between the two methods .

Data Fabric It is a design concept and architecture method , It aims to solve the complexity of data management , Minimize interference to data users , Make sure anywhere 、 Any data on any platform can be effectively accessed .Data Fabric It is essentially a metadata driven approach , Both AL/ML Drive enhancement , And contains cloud primitives 、 Microservices 、API Drive, etc , Used to link different data toolsets . In an increasingly isomerized environment ,Data Fabric It is very important to see the emergence of new technology . Because at this moment , The problem of data diversity is becoming more serious .

Data Mesh In solving problems and Data Fabric Very similar , That is, the problem of managing data in heterogeneous data environment . But the difference between the two is ,Data Mesh Allow distributed teams to manage data in their own way while respecting common governance rules , and Data Fabric It is to build a single virtual management layer on top of distributed data .Data Mesh Hope to correct the inconsistency between data lake and data warehouse .

To sum up one more level ,Data Mesh Focus on organizational change , It focuses on people and processes , Not Architecture , and Data Fabric Technology centric , It is an architectural approach , It handles the complexity of data and metadata in an intelligent way , And can work together well . There is no conflict between the two , It can even collaborate effectively , You can think of them as frameworks rather than architectures .

Previously mentioned data lake and data warehouse , Actually at the moment , How to provide the best data storage for data analysis needs has always been a hot topic , The competition of related products is fierce . Data warehouse and data lake have always been the most widely used big data storage architecture , In recent years, hucang integrated , It claims to combine the flexibility of data lake and the convenience of data management of data warehouse , But so far , There are few best practices in the industry , Numerous marketing .

Data Lake vs Data warehouse  vs The discussion on the integration of lake and warehouse will continue for a long time , Choose which architecture , It depends on the type of data you are dealing with 、 Data source and data usage .

We want to find best practices , For your reference . Therefore, it will be on 7 month 31 Japan -8 month 1 Day QCon Global software development conference （ Guangzhou Railway Station ） Specially planned 「 Modern data architecture selection 」 project , Integrate the lake and warehouse 、Flink Latest updates 、Data Fabric、Data Mesh Relevant practices of are gathered here , I hope it is helpful for your selection .

QCon The schedule of the global software development conference Guangzhou station has been launched on the official website ,50+ Technical practice cases are publicly shared for the first time , Click on the bottom 【 Read the original 】 A detailed speech outline for the overview topic . The limited time ticket discount is coming to an end , Cutting edge case sharing cannot be missed . Interested students contact the ticket manager to register ：15600537884（ Same as wechat ）～