Typical data Lake application cases

Typical data Lake application cases

1.  Advertising data analysis

In recent years , The cost of traffic acquisition is getting higher and higher , The doubling of online channel customer acquisition cost makes all walks of life face severe challenges . In the context of the rising cost of Internet advertising , The business strategy of spending money to buy new traffic is bound to fail . The optimization of traffic front end has become the end of a powerful crossbow , Use data tools to improve the target conversion of traffic after arriving at the station , Refine all links of advertising operation , Is a more direct and effective way to change the status quo . At the end of the day , To improve the conversion rate of advertising traffic , Must rely on big data analysis .

In order to provide more decision support , More buried point data collection and analysis are needed , Including but not limited to channels 、 Launch time 、 Crowd drop , Take the click through rate as the data index for data analysis , So as to give a better 、 More rapid solutions and suggestions , Achieve high efficiency and high output . therefore , In the face of multi-dimensional advertising 、 multimedia 、 Multiple advertising spaces and other structured 、 Semi structured and unstructured data collection 、 Storage 、 Requirements such as analysis and decision-making suggestions , The data Lake analysis product solution has been warmly favored by advertisers or publishers in the selection of new generation technologies .

DG Is a global leading enterprise international intelligent marketing service provider , Based on advanced advertising technology 、 Big data and operational capabilities , Provide customers with global high-quality user acquisition and flow realization Services .DG From the beginning of its establishment, it has decided to build its... Based on the public cloud IT infrastructure , first DG I chose AWS Cloud platform , Mainly use its advertising data in S3 It is stored in the form of a data lake , adopt Athena Conduct interactive analysis . However, with the rapid development of Internet advertising , The advertising industry has brought several challenges , Several key problems must be solved in the publishing and tracking system of mobile advertising ：

• Concurrency and peak problems . In the advertising industry , Traffic peaks often occur , Instant hits may reach tens of thousands , Even hundreds of thousands , This requires the system to have very good scalability to quickly respond to and process every click

• How to realize the real-time analysis of massive data . In order to monitor the advertising effect , The system needs to analyze each click and activation data of the user in real time , At the same time, the relevant data is transmitted to the downstream media ;

• The amount of data on the platform is growing rapidly , Daily business log data is continuously generated and uploaded , Exposure 、 Click on 、 The pushed data is being processed continuously , The amount of data added every day is already 10-50TB about , It puts forward higher requirements for the whole data processing system . How to efficiently complete the offline of advertising data / Near real time statistics , Conduct aggregation analysis according to the dimensional requirements of advertisers .

In view of the above three business challenges , meanwhile DG The daily incremental data of this customer is rapidly increasing （ The current daily data scanning volume has reached 100+TB）, Continue to AWS Platform usage encountered Athena Read S3 Data bandwidth bottleneck 、 The lag time of data analysis is getting longer and longer 、 Rapidly rising input costs in response to the growth of data and analysis demand , After careful 、 Careful testing and Analysis , The final decision was to start with AWS Move all cloud platforms to Alibaba cloud platform , The composition of the new frame is as follows ：

from AWS After moving to Alibaba cloud , We designed... For this customer “ utilize Data Lake Analytics + OSS” Extreme analytical ability to deal with business peaks and troughs . On the one hand, it is easy to deal with the temporary analysis from brand customers . On the other hand, use Data Lake Analytics Powerful computing power , Analysis by month 、 Quarterly advertising , Calculate exactly how many activities there will be under a brand , Each activity is divided into media , Sub market , Sub channel , branch DMP The delivery effect , Further enhance the sales conversion rate brought by Jiahe intelligent traffic platform for brand marketing .

And in the total cost of ownership of advertising and Analysis ,Data Lake Analytics Provided Serverless Our flexible service is on demand , No need to buy fixed resources , The tide of business brings about fluctuations in resources , Meet the needs of elastic analysis , At the same time, it greatly reduces the operation and maintenance costs and use costs .

Overall ,DG from AWS After switching to alicloud , It greatly saves the hardware cost 、 Labor cost and development cost . As a result of DLA serverless The cloud service ,DG There is no need to invest a lot of money to buy servers in advance 、 Storage and other hardware devices , There is no need to buy a large number of cloud services at one time , The scale of its infrastructure is completely on-demand ： Increase the number of services when the demand is high , Reduce the number of services when demand decreases , Improve the utilization of funds .

The second significant benefit of using alicloud platform is the improvement of performance . stay DG The rapid growth period of services and the subsequent access period of multiple service lines ,DG The number of visits to mobile advertising systems often grows explosively , But originally AWS The scheme and platform are in Athena Read S3 Data has encountered a great bottleneck in data reading bandwidth , The time of data analysis is getting longer and longer , Alibaba cloud DLA union OSS The team has made great optimization and transformation , meanwhile ,DLA Database analysis on the computing engine （ And TPC-DS Number one in the world AnalyticDB Shared computing engine ） Than Presto The capability of native computing engine is improved dozens of times , Also great for DG Improved analysis performance .

2.  Game operation analysis

Data lake is a kind of TCO Excellent big data infrastructure . For many fast-growing game companies , A popular game , Often in the short term, the relevant data grow very fast ; meanwhile , The technology stack of the company's R & D personnel is difficult to match the increment and growth rate of data in the short term ; here , Explosive growth of data is difficult to use effectively . Data lake is a technical choice to solve such problems .

YJ Is a fast-growing game company , The company hopes to conduct in-depth analysis based on relevant user behavior data , Guide the development and operation of the game . The core logic behind data analysis is that with the expansion of market competition in the game industry , Players have higher and higher requirements for quality , The life cycle of game projects is getting shorter and shorter , Directly affect the input-output ratio of the project , Data operation can effectively prolong the life cycle of the project , Accurately control the business trend of each stage .

With the increasing cost of traffic , How to build an economy 、 Efficient and refined data operation system , To better support business development , Also become more and more important . The data operation system needs its supporting infrastructure , How to choose such infrastructure , It is a problem that the company's technology decision-makers need to think about . The starting point of thinking includes ：

• Be flexible enough . For the game , It often erupts in a short time , The amount of data is surging ; therefore , Whether it can adapt to the explosive growth of data , Meeting elastic demand is a key consideration ; Whether it's computing or storage , All need to have enough elasticity .

• Have enough cost performance . For user behavior data , It often takes a long period to analyze and compare , Such as retention rate , In many cases, we need to consider 90 God even 180 Day customer retention rate ; therefore , How to store massive data in the most cost-effective way for a long time is a key consideration .

• Have enough analytical skills , And scalable . In many cases , User behavior is reflected in buried point data , The buried point data needs to be consistent with the user registration information 、 Landing information 、 Correlation analysis of structured data such as bills ; therefore , In data analysis , At least big data is needed ETL Ability 、 The access ability of heterogeneous data sources and the modeling ability of complex analysis .

• To match the company's existing technology stack , And the follow-up is conducive to recruitment . about YJ, An important point in technology selection is the technical stack of its technicians ,YJ Most of our technical teams are only familiar with traditional database development , namely MySQL; And the hands are tight , The only technicians who do data operation analysis are 1 individual , There is no ability to independently build the infrastructure of big data analysis in a short time . from YJ From the angle of , It is best that most of the analysis can pass SQL complete ; And in the recruitment market ,SQL The number of developers is also much higher than that of big data development engineers . For the customer's situation , We helped our customers transform their existing solutions .

Before transformation , All structured data of customers are in a high specification MySQL Inside ; The player behavior data is through LogTail Collect to log service （SLS） in , Then, it is delivered from the log service to OSS and ES in . The problem with this architecture is ：

• Behavioral data and structured data are completely separated , Unable to perform linkage analysis ;

• Provide intelligent retrieval function for behavioral data , Unable to do deep mining analysis ;

• OSS Used only as a data storage resource , Not mining enough data value .

in fact , We analyze that the customer's existing architecture actually has the prototype of the data Lake ： The full data is already in OSS Saved in , Now we need to further supplement the customer's requirements for OSS Ability to analyze data in . And the data lake is based on SQL The data processing mode also meets the needs of customers for developing technology stack . Sum up , We have made the following adjustments to the customer's architecture , Help customers build a data Lake .

Overall , We didn't change the customer's data link , It's just OSS On the basis of , Added DLA Components , Yes OSS The data is processed twice .DLA Provides standards SQL Calculation engine , At the same time, it supports access to various heterogeneous data sources . be based on DLA Yes OSS After processing the data , Generate data directly available to the business . however DLA The problem is that it cannot support interactive analysis scenarios with low latency requirements , To solve this problem , We introduced cloud native data warehouse ADB To solve the delay problem of interactive analysis ; meanwhile , Introduce... At the front end QuickBI As a visual analysis tool for customers .YJ The scheme is figure 14 The lake Warehouse Integration Solution shown is a classic landing case in the game industry .

YM Is a data intelligence service provider , Provide a series of data analysis and operation services for all kinds of small and medium-sized businesses . The specific technical logic is shown in the figure below .

The platform side provides multi terminal services SDK For users （ Businesses provide web pages 、APP、 Applet and other access forms ） Access various buried point data , The platform side takes SaaS Provide unified data access services and data analysis services in the form of . Businesses access various data analysis services to conduct more fine-grained buried point data analysis , Complete behavior statistics 、 Customer portrait 、 Customers circle 、 Advertising monitoring and other basic analysis functions . However , such SaaS In mode , There will be some problems ：

• Due to the diversification of business types and needs , Provided by the platform SaaS Class analysis is difficult to cover all types of businesses , Unable to meet the customized needs of merchants ; For example, some businesses pay attention to sales , Some focus on customer operations , Some focus on cost optimization , It's hard to meet all the needs .

• For some advanced analysis functions , For example, customers who rely on custom labels circle 、 Customer defined extension and other functions , Unified data analysis services can not meet ; In particular, some custom tags depend on the merchant's custom algorithm , Unable to meet customers' advanced analysis needs .

• Data asset management requirements . In the age of big data , Data is an enterprise / The assets of the organization have become a consensus , How to make the data belonging to the merchant reasonable 、 Long term precipitation , It's also SaaS Service considerations .

Sum up , We introduce the data lake model on the basic model in the figure above , Let the data Lake serve as the merchant's precipitation data 、 Output model 、 Analyze the infrastructure supporting the operation . After introducing the data Lake SaaS The data intelligence service mode is as follows .

Pictured 21 Shown , The platform provides one click Lake building service for each user , Businesses use this function to build their own data lake ,“ Build a lake with one click ” On the one hand, it can help businesses build a data model of all buried point data （schema） Sync to data Lake ; On the other hand , Synchronize all buried point data belonging to the merchant to the data lake , And based on “T+1” The pattern of , Archive daily incremental data into the lake . The service model based on data lake is based on the traditional data analysis service , It gives users data capitalization 、 Analyze the three capabilities of modeling and service customization ：

• Data capitalization capability . Using data Lake , Businesses can continuously precipitate their own data , How long is the data kept , How much does it cost , It is entirely up to the merchant to decide . The data lake also provides data asset management capabilities , In addition to managing raw data , It can also save the processed process data and result data by category , Greatly enhance the value of buried point data .

• Analytical modeling capability . There is more than raw data in the data Lake , And the model of buried point data （schema）. The buried point data model reflects the abstraction of business logic by the global data intelligent service platform , Through the data lake , In addition to exporting raw data as assets , The data model is also output , With the help of buried point data model , Businesses can have a deeper understanding of the user behavior logic behind the buried point data , Help businesses better insight into customer behavior , Get user needs .

• Service customization capability . With the help of the data integration and data development capabilities provided by the data Lake , Based on the understanding of buried point data model , Businesses can customize the data processing process , The original data is processed iteratively , Extract valuable information from data , Finally get the value beyond the original data analysis services .