Xiaodi class massive data processing business short chain platform

One 、 Massive data lesson + Introduction to architecture course

• Massive data projects , It takes little time to prepare for class 1 Comprehensive practical projects in , from 0~1 Developers use short chain platforms ;

• Start from the customization of product requirements documents , Master the core points and professional terms of Internet manufacturers , Project cycle and background

• Adopt the new version of microservice architecture + Container arrangement + Massive data processing + Big data technology stack + Distributed scheduling tracking

• Distributed transactions + Data consistency processing + Distributed lock solutions

• master Flink Real time computing + Big data is cleaned in layers +ClickHouse actual combat

• A new generation of container orchestration platform Rancher2.5 Deploy multiple nodes + Dynamic capacity expansion practice + The container arrangement is rapidly recognized

• From a macro point of view, there are multiple middleware and JDK Source code design ideas , And then apply it in the project

• Understand that you like to ask about the design idea of the bottom layer in the interview of a large factory , And ordinary companies can't

• From an architect's point of view, we can draw ideas ： Choose multiple solutions ,【 Some will be implemented in the project , Part of the analogy 】

• Redis6.X Lazy deletion Apply in the massive data traffic package

• ConcurrentHashMap The must kill skill of subsection lock in the second kill

• CopyOnWriteArrayList Read / write separation + The essence of the final consensus

• Kafka sender and main Threads send and... Asynchronously liner.ms Application scenarios

• Game confrontation between technical solutions and product requirements

• … more

Two 、 Large class project Corresponding position and ability

• technology ： On the ali P7 Architect competency level + Intersection of big data technology stacks

• management ： belt 5 To 10 Human team + Assessment of

• Salary ： Annual general contracting of first tier cities 40 To 70 ten thousand 、 a monthly salary 25 To 40k

• Master product analysis 、 Solution 、 Team management methodology , Report to the superior for communication + Assess the management of the lower level

3、 ... and 、 Short chain platform technology stack

• Microservice technology ： new edition SpringCloudAlibaba Family bucket +SpringBoot2.5+Nacos2.x+MybatisPlus

• cache +MQ middleware + Dispatch ：Redis6.X+RabbitMQ+Zookeeper+Kafka + XXLJob

• Massive data is divided into databases and tables + File store ：Mysql8.0+ShardingSphere Multi dimension database and table + Alibaba cloud OSS

• Real time computing + Data processing + Storage Visualization ：Flink1.13 + ClickHouse + HDFS + Data cleaning and layering + Echart Visualization data

• Distributed Link Tracking + monitor + Persistent storage ：Apache Skywalking + ElasticSearch7.X

• Online deployment ：Jenkins CICD + Alibaba cloud Git Warehouse + Alibaba cloud ECS CentOS7 + Rancher2.5 + Container scheduling

• Advanced test must have production environment flow pressure test ：Jmeter5.X Single interface pressure measurement + Mixed traffic recording + Expansion and replay of flow pressure measurement

Four 、 Highlights of the massive data commercial short chain platform project

Massive data - The commercial short chain platform starts from 0~1 actual combat （P1、P2、P3 The difficulty decreases gradually ）

Account micro service

Account module

Graphic verification code anti brush design （P2）

SMS verification code access (P2)

Free traffic packet processing for new users under high concurrency （P1）

Distributed file storage （P2）

Traffic packet module ( a key ： Multi scenario distributed transaction + lock 、 Data consistency 、 Business complex )

Traffic packet usage management （P1）

A solution for processing traffic packet expiration under massive data ： inert + Random （P1）

Daily traffic packet update solution under massive data ： inert （P1）

Packet deduction processing for high parallel traffic （P1）

High concurrency 、 Multi case distributed transaction + lock 、 Complex business logic （P1）

Hot and cold data archiving and processing （P1）

Commodity micro service

Commodity module

Product list and details （P3）

Payment module ( a key )

AOP application + Custom annotation anti resubmission design （P1）

New wechat code scanning payment V3 Interface document docking （P1）

PC End Alipay payment code scanning docking （P2）

Multi channel payment callback notification + asynchronous MQ Undertaking (P1)

ShardingSphere from 0 To 1 To the actual combat of commercial projects （P1）

Order data snapshot + Regular sub database and sub table （P3）

After the sub-library sub-table C End parsing and B End query processing solution 1 :NoSQL Law （P1）

Short chain service

Short chain module

Multi case short chain code design scheme and landing （P1）

High performance short chain analysis （P1）

Multi case data reporting design and scheme comparison （P1）

C End parsing +B End query Common solutions for sub database and sub table （P1）

Massive data processing by database and table （P1）

After the sub-library sub-table C End parsing and B End query processing solution 2 ： Redundant double writing （P1）

Short chain after dividing database and table CURD【 Redundant double write 】 Distributed transaction data consistency solution （P1）

A single PartitionKey Make solution 3 of database and table division ： Gene method （P1）

Short chain library table database 【 Migration free expansion 】 Solution 4 ： Gene method (P1)

Data visualization services

from 0~1 master ClickHouse new generation OLAP database （P2）

from 0~1 Master the new version of distributed scheduling tasks xxl-job+alibabcloud actual combat （P2）

ClickHouse Multi scenario query 、echart Visual data docking （P1）

Dynamic blacklist detection solution for illegal short chain sites （P1）

Real time computing services

Back end engineers view advanced data warehouse + layered + Technical vision in big data field （P1）

Flink+Kafka Real time calculation of short chain data + Multidimensional data processing （P1）

Async Asynchronous Association query + Multi dimension wide table expansion （P1）

Flink Multi stream merge +dws Layer integration ClickHouse Storage （P1）

DevOps The whole link process + New generation container orchestration platform deployment （P1、P2、P3 The difficulty decreases gradually ）

Jenkins CICD build + Alibaba cloud Git Warehouse +Docker Private image warehouse construction and deployment (P1)

Rancher2.x Deploy + Quick start + Get started with the core knowledge of container choreography + Deploy Linux Public cloud （P1）

Skywalking Link tracking +SpringCloud Containerized integration +ElasticSearch7.X Persistent deployment (P2)

Jmeter5.x Pressure measurement + Mixed traffic recording + Expansion and replay of flow pressure measurement （P1）

report 、 analysis 、 Communication soft skills + Interview job hopping resume writing

Technology selection / Team management methodology + Inter team responsibilities of Internet companies are explained in cooperation (P1)

Requirement document analysis / Explanation of commercialization ability （P2）