Data warehouse 4.0 notes - business data collection

1 Introduction to e-commerce business

 SKU = Stock Keeping Unit（ The basic unit of inventory ）. Now it has been extended to the abbreviation of product unified number , Each product has a unique SKU Number .

 SPU（Standard Product Unit）： yes The smallest unit of commodity information aggregation , It's a group. Reusable 、 Easy to retrieve A standardized set of information .

for example ：iPhoneX Cell phones are SPU. A silver one 、128G In memory 、 Support for Unicom Network iPhoneX, Namely SKU.

2 Business data acquisition module

Core operations ： take MySQL The business data on is synchronized to HDFS above （ Batch synchronization and real-time synchronization ）

MySQL install

Install the package and JDBC The driver is uploaded to /opt/software, total 6 individual

[[email protected] ~]$ cd /opt/software/

Too many compressed packages , Sort it out ：

[[email protected] software]$ mkdir flume
[[email protected] software]$ mkdir zookeeper
[[email protected] software]$ mkdir java
[[email protected] software]$ mkdir kafka
[[email protected] software]$ mkdir mysql

[[email protected] software]$ mkdir hadoop
[[email protected] software]$ ll

[[email protected] software]$ mv apache-flume-1.9.0-bin.tar.gz flume/
[[email protected] software]$ mv apache-zookeeper-3.5.7-bin.tar.gz zookeeper/
[[email protected] software]$ mv hadoop-3.1.3.tar.gz hadoop
[[email protected] software]$ mv jdk-8u212-linux-x64.tar.gz java/
[[email protected] software]$ mv kafka_2.11-2.4.1.tgz kafka
[[email protected] software]$ ll

  Delete all the unwanted documents

  Get into MySQL, Upload MySQL Installation package required

[[email protected] software]$ cd mysql/

Unload the Mysql-libs（ If you've installed it before MySQL, I want to unload them all ）

[[email protected] mysql]$ rpm -qa | grep -i -E mysql\|mariadb | xargs -n1 sudo rpm -e --nodeps

[[email protected] mysql]$ rpm -qa | grep -i -E mysql\|mariadb

  If it is an alicloud server, follow the steps below

explain ： Because Alibaba cloud server is installed with Linux Minimum system version , There are no following tools , So you need to install .

（1） uninstall MySQL rely on , Although the machine is not equipped with MySQL, But this step is indispensable

[[email protected] software]# sudo yum remove mysql-libs

（2） Download dependencies and install

[[email protected] software]# sudo yum install libaio

[[email protected] software]# sudo yum -y install autoconf 

k Start installation ：

[[email protected] mysql]$ rpm -qa | grep -i -E mysql\|mariadb | xargs -n1 sudo rpm -e --nodeps

[[email protected] mysql]$ sudo rpm -ivh 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm

[[email protected] mysql]$ sudo rpm -ivh 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm

[[email protected] mysql]$ sudo rpm -ivh 04_mysql-community-client-5.7.16-1.el7.x86_64.rp

[[email protected] mysql]$ sudo rpm -ivh 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm

start-up MySQL

[[email protected] software]$ sudo systemctl start mysqld

Check the status [[email protected] mysql]$ sudo systemctl status mysqld

see MySQL password

[[email protected] software]$ sudo cat /var/log/mysqld.log | grep password

Enter with the password you just found MySQL（ If you make a mistake , Put single quotation marks on the password ）

 [[email protected] mysql]$ mysql -uroot -p'f&8U;US.yhP#'

Set complex password （ because MySQL Password policy , This password must be complex enough ）

mysql> set password=password("Qs23=zs32");

change MySQL Password policy

mysql> set global validate_password_length=4;

mysql> set global validate_password_policy=0;

Set a simple and easy to remember password

mysql> set password=password("000000");

Configure remote login ： Get into MySQL library

mysql> use mysql

Inquire about user surface

mysql> select user, host from user;

modify user surface , hold Host The content of the table is changed to %

mysql> update user set host="%" where user="root";

Refresh

mysql> flush privileges;

sign out

mysql> quit;

Business data generation

Connect MySQL

Use one MySQL The client generates a database （Navicat for MySQL）

 

Test the connection first , Successful connection , To determine the .（ My host uses hadoop102, The connection fails , Switch to IP The address is successfully connected , I don't know which link went wrong , Always use IP）

 

 

  Start execution

 

 

  Generating business data

stay hadoop102 Of /opt/module/ Create under directory db_log Folder

[[email protected] module]$ mkdir db_log/

 

 [[email protected] db_log]$ vim application.properties

 

 

[[email protected] db_log]$ java -jar gmall2020-mock-db-2021-01-22.jar

  Write data complete .

see gmall database , To see if there is 2020-06-14 There's a lot of data coming up

  Generated 2020-06-14 The content of , If you want to reproduce the data of other days , Go straight in application.properties Modification time , also 1 Set as 0.