Risk control system, implemented by flink+clickhouse!

599 A special price of yuan   The original price  1899 element   Of

《 Practice of enterprise level risk control system project of Internet manufacturers 》

=== Course is an introduction to  ===

| SparkStreaming  Enterprise level project practice - Financial risk control

Risk control level of risk control , In the actual project, it is divided into two levels: warning and risk . When the risk level is reached , It will trigger the mechanism of sending text messages to remind users .

• Business explanation based on user login and operation rules
  

• Business process development

• SparkStreaming Integrate 0.8 kafka How to solve the problem of data loss in the production environment

• SparkStreaming Integrate kafka0.10+ How to solve the problem of data loss in the production environment

• Interview topics ： How to integrate into your resume , Speech arrangement

| Project iteration

be based on Flink Realize risk control logic , Business description , Introduction to data sources , Prepared data source ( Behavioral data , Image data ), The first 1 Implementation of version requirements , Login rule processing and mobile number modification event processing

| Structure optimization

Encapsulate rule processing logic , Code optimization query service class module extraction , Definition of portrait data structure , everyday 2000+ How to optimize the transmission efficiency of 100 million pieces of data

• Double current Join Analysis of the image matching service module implementation and testing , Dynamic conditional encapsulation

• User tag feature data portrait system docking
  

• User portrait concept 、 Project overview and platform testing , Data source preparation
  

• User basic data model development , Tag building
  

• Rule matching label model development

• Rule matching label model development

• Search tag model development

• Mining label algorithm model construction

• Portrait label index 、 Classification recommendation and multi data source integration

• The first 2 Version iteration , Step matching logic algorithm improvement , Problems caused by state storage of a large amount of data and solutions , Use clickhouse Solve the problem of event data storage

• ClickHouse Table engine definition integration kafka,ClickHouse Integrate kafka data ,ClickHouse Query service SQL Realization , Query service implementation interface design

• The first 3 Version core logic optimization , Time span data utilization FlinkState and clickhouse Advantages and disadvantages of storage , Query routing module architecture design , Module encapsulation

• Two way merge query service implementation , The first 3 Analysis of performance problems of version scheme storage , Introduce cache solution clickhouse The concurrent pressure of

• Technology selection of cache system hbase_VS_redis_VS_State Comparison , Cache data model design , Cache module core logic design

• Design of query time dividing point , User profile cache design effectiveness , Rule cache module development , Cache module test .

• The disadvantage that rules cannot be dynamically matched , How to design dynamic rules , Risk control rule management is upgraded to dynamic configuration

• flink Integrate drools Realize dynamic management of rules ,drools Basic practice of template engine , Syntax parsing .

• Flink Integrate Canal Design ,DRL File input kafka, adopt canal Listen to the mysql To implement the definition ,FlinkCDC Scheme introduction ,FLinkCDC The advantages of

• Rule matching core logic design , Rule parameter entity design , Rules match the overall process design , Core computing router Design , Two way query mechanism design

• Flink Dynamic loading rules ,Drools Rule storage table design , Rule management needs to be clear , The rules state Storage design

• Rule management processor development ,Flink Call the core function development of dynamic rules

• Dynamic rules DRL Code development , Overall structural design ,Sql Code development

• DRL And SQL Template engine design and development , Design and development of monitoring and analysis module of risk control system , The whole process design of rule dynamic injection

• The rule matching module is upgraded online , Debugging test , The idea of dynamic compilation and loading of project query core services

• Model based risk control scenario needs explanation

• Feature selection of Feature Engineering

• Feature realization of Feature Engineering

• Model prediction algorithm implementation

| Solution

• end-to-end Data strong sequential solution ： analysis flink How to ensure the sequence of messages in the process of data flow , Analyze which scenarios may cause data loss due to retry. Data sequencing , Analyze the solution of message sequence calculation

• end-to-end exactly once Solution ： analysis flink Of exactly once How to guarantee the semantics of computing once and only once . analyse flink Bottom exactly once principle

• late arrive Automatic error correction solution ： Analysis in real-time calculation , Some data of the day did not reach the calculation , The next day late arrive After delayed arrival , How to automatically complete error correction calculation

• Enterprise level data monitoring and offline data error correction solutions ： Analyze how to develop a set of offline computing links for the data source ,T+1 The offline calculation results are compared with the real-time calculation results , If an exception is found in offline detection , Make offline error correction compensation

• flink How to recover automatically after a computing node crashes 、redis High availability architecture design 、redis After the cluster crash flink Native cache degradation scheme 、clickhouse High availability Architecture 、clikchouse Degradation calculation scheme after crash

| Interview topics ： How to write a resume for the project , Expression script, etc

=== Course content ===

（* The following is only part of the course content , See the end of the article for a detailed outline ）

FlinkSQL Stream batch integrated processing

=== About Instructor ===

• Many years of experience in the architecture of first-line Internet manufacturers , Led the design and development of many large-scale Internet projects

• Output a large number of original technical dry goods articles , The reading volume of the whole network is 10000 +

=== Lessons learned  ===

• Demand analysis and detailed design of risk control project

• be based on Flink and Clickhouse Realize the function of user portrait

• Flink Node crash 、 Cache degradation and other production level fault analysis

=== Course benefits ===

Fukuichi

In order to let more students learn and earn ,《 Practice of enterprise level risk control system project of Internet manufacturers 》 The course only needs 599 element , Great cost ! You can scan the code to buy , First come first served basis ！

Welfare II

No second kill to 《 Practice of enterprise level risk control system project of Internet manufacturers 》 Students in the course , You can participate in the group activities of this course , Two people only need  579 element , Three people only need  559 element .

=== Syllabus ===

Scroll to see more

===  Course entrance ===

Long press to scan the QR code below , Direct to the course column

* Second kill notice ：

1.  Students of the Confucian ape architecture class who have purchased this course , No need to buy again

2.   After purchase, watch it on the advanced player （ Theft prevention ）, Support PC End and mobile phone end