Data Lake: introduction to delta Lake

1. DeltaLake What is it?

        Delta Lake yes DataBricks Company open source 、 Storage framework for building Lake Warehouse Architecture . Able to support Spark,Flink,Hive,PrestoDB,Trino Etc / Calculation engine . As an open format storage layer , It provides batch flow integration at the same time , Provide reliable support for the lake warehouse structure , Safe , High performance guarantee .

 

2. Key features         

Delta Lake Key features ： 

1. ACID Business ： Through different levels of isolation strategies ,Delta Lake Support multiple pipeline Concurrent read and write ;
2. Data version management ：Delta Lake adopt Snapshot And so on 、 Version of audit data and metadata , And then support time-travel Query the historical version data or trace back to the historical version ;
3. Open source file format ：Delta Lake adopt parquet Format to store data , In order to achieve high-performance compression and other characteristics ;
4. Batch flow integration ：Delta Lake Support batch and streaming data reading and writing ;
5. Metadata evolution ：Delta Lake Allow users to merge schema Or rewrite schema, To adapt to the changes of data structure in different periods ;
6. rich DML：Delta Lake Support Upsert,Delete And Merge To meet the needs of users in different scenarios , such as CDC scene ;
7. File structure ： Lake watch is more than ordinary Hive One big difference between tables is that ： The metadata of the lake table is self managed , Stored in the file system .

2. Document organization

         The following figure for Delta Lake Table file structure ：

 

Delta Lake The file structure of is mainly composed of two parts ：

1. _delta_log Catalog ： Storage deltalake All metadata information of the table , among ：

• Each operation on the table is called commit, Including data operation （Insert/Update/Delete/Merge） And metadata operations （ Add new column / Modify table configuration ）, Every time commit Will generate a new json Format log file , Record this time commit The behavior produced on the table （action）, Such as adding new files , Delete file , Updated metadata information, etc ;
• By default , Every time 10 Time commit Will automatically merge into one parquet Format checkpoint file , Used to speed up the parsing of metadata , And metadata files that support periodic cleaning of history ;

        2. Data directory / file ： except _delta_log Outside the directory is the file that actually stores the table data ; We need to pay attention to ：

1. DeltaLake The data organization of partitioned tables is the same as that of ordinary tables Hive surface , Partition fields and their corresponding values are part of the actual data path ;
2. Not all visible data files are valid ;DeltaLake In order to snapshot Form organization table , newest snopshot The corresponding valid data file is in _delta_log Manage in metadata ;

3. Metadata mechanism

        Delta Lake adopt snapshot To manage multiple versions of tables , It also supports the of historical versions Time-Travel Inquire about . Whether it's querying the latest snapshot Or a version of history snapshot Information , You need to parse to get the corresponding snapshot Metadata information , Mainly involves ：

• At present DeltaLake The read-write version of the protocol （Protocol）;
• Field information and configuration information of the table （Metadata）;
• List of valid data files ; This is done through a set of new files （AddFile） And delete files （RemoveFile） To describe ;

         That's loading concrete snopshot when , To speed up the loading process , Try to find a version less than or equal to this version first checkpoint file , Then combine the following until the current version of log file , Analyze and get metadata information together .

Reference material

[01] https://www.cnblogs.com/yunqishequ/p/16265824.html