Two architectures of data integration: ELT and ETL

Two architectures for data integration ：ELT and ETL

Integration is one of the tasks of data engineers . Generally speaking , The work of Data Engineer includes data ETL And the implementation of data mining algorithm . Algorithm implementation   You can understand , Through data mining algorithm , Find it in the data warehouse “ gold   What is? ETL Well ？ ETL It's English Extract. Transform and Load Abbreviation , As the name suggests, it includes data extraction 、 transformation 、 Load three processes .ETL can   So it's before data mining " Prepare vegetables " The process .

Let me explain data extraction 、 transformation 、 Load these three processes .

ETL The process is extraction （Extract）—— transformation （Transform）—— load （Load）, After the data source is extracted, the transformation is carried out first , Then write the conversion result to   Destination .

ELT The process is to extract （Extract）— load （Load）—— Transformation （Transform）, After extraction, write the result to the destination first , Then use the aggregation of database   Combined with analytical capabilities or external computing frameworks , Such as Spark To complete the conversion steps

At present, the mainstream architecture of data integration is ETL , But it will be used in the future ELT As a data integration architecture, there will be more and more . Doing so will bring many benefits :

ELT and ETL comparison , The big difference " Re extract and load , Light conversion ", Thus, a data integration platform can be built with a lighter scheme . Use ELT  Method , After the extraction is complete , Data loading starts immediately . On the one hand, it saves time , On the other hand ELT allow BI The analyst has unlimited access to the entire raw number   According to the , It provides analysts with greater flexibility , To better support the business .