CDH cluster installation

Environmental statement

Cloudera Manager：5.13.3
CDH: 5.13.3
CentOS 7.3  x64
JDK 1.8.0_152
MySQL 5.7.20
 Close the firewall and SELinux ( All nodes )
 Configure time synchronization （ All nodes ）

# download CM Installation package ( Master node )
wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1
wget http://archive.cloudera.com/cdh5/parcels/5.13.3/manifest.json

#Mysql Installation package and JDBC drive ( Master node ) During production installation, only the driver needs to be installed 
wget http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.45.tar.gz

One 、 install

1、 Install dependency packages

yum install -y  psmic cmake gcc gcc-c++ ncurses-devel bison   libxml2 libxml2-devel  bzip2-devel  libssl* python-devel  libaio* bzip2 libxml2-python
yum install -y chkconfig bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
yum install -y make automake autoconf libtool perl  mod_ssl  python-psycopg2  postgresql-server openssl-devel MySQL-python

Two 、 Configuration environment

1. The network configuration ( All nodes )

# Modify hostname ,  Remember that host names cannot be underlined 
# Modify hostname (/etc/hostname):
hostnamectl set-hostname  cdh-master
hostnamectl set-hostname  cdh-slave1
hostnamectl set-hostname  cdh-slave2
hostnamectl set-hostname  cdh-slave3
hostnamectl set-hostname  cdh-slave4
hostnamectl set-hostname  cdh-slave5

# modify  /etc/sysconfig/netroot,  Make it effective  source
vim /etc/sysconfig/netroot    
NETROOTING=yes
HOSTNAME=cdh-xxx

# modify  /etc/hosts  Join in 
vim /etc/hosts
192.168.168.83 cdh-master
192.168.168.84 cdh-slave1
192.168.168.85 cdh-slave2
192.168.168.86 cdh-slave3
192.168.168.87 cdh-slave4
192.168.168.88 cdh-slave5

2. Get through SSH, Set up ssh Login without password , Nodes in the cluster can access each other without passwords ( All nodes )

ssh-keygen 
ssh-copy-id -i id_rsa.pub [email protected]*

3. install jdk（ All nodes ）

mkdir /usr/java
tar zxf /root/jdk1.8.0_111.tar.gz -C /usr/java

# Configure environment variables 
vim /etc/profile
# Add the following :
JAVA_HOME=/usr/java/jdk1.8.0_111
JRE_HOME=/usr/java/jdk1.8.0_111/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH

# Refresh configuration 
source /etc/profile

4. Installation configuration MySql（ Master node ）

(1) Unzip the installation package

tar zxf mysql-5.6.39-linux-glibc2.12-x86_64.tar.gz -C /opt/

(2) Modify the file directory name

mv /opt/mysql-5.6.39-linux-glibc2.12-x86_64/ mysql

(3) Create a data directory

mkdir sqldata
useradd -M -s /sbin/nologin mysql
# to grant authorization 
cd /data/
chown -R mysql.mysql mysql

(4) Configure environment variables

vim  /etc/profile
PATH=$PATH:/opt/mysql/bin
source /etc/profile

(5) initialization

yum install libaio* -y
scripts/mysql_install_db --user=root --basedir=/opt/mysql --datadir=/opt/mysql/sqldata
chmod -R 777 sqldata

(6) Copy profile

cp support-files/my-default.cnf /etc/my.cnf
vim /etc/my.cnf
max_allowed_packet=1024M
basedir = /opt/mysql/
datadir = /opt/mysql/sqldata

(7) Copy startup script

cp support-files/mysql.server /etc/init.d/mysqld
chkconfig --add mysqld
chkconfig mysqld on
service mysqld start

(8) Sign in

mysqladmin -uroot  password 123abcABC
mysql -uroot -p123abcABC

(9) Authorized user root Use password passwd Connect from any host to mysql The server

grant all privileges on *.* to 'root'@'%' identified by '123abcABC' with grant option;
flush privileges;

(10) To build database （ For installation of rear components ）

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

3、 ... and 、 install cdh

1. install Cloudera Manager Server and Agent

mkdir /opt/cloudera-manager
tar zxf cloudera-manager-centos7-cm5.13.3_x86_64.tar.gz
mv cloudera  cm-5.13.3  /opt/cloudera-manager
mkdir /var/cloudera-scm-server（ All nodes ）

2、Mysql Drive on the cm Of lib Next ( All nodes )

scp mysql-connector-java-5.1.45.tar.gz [email protected]*:/root
tar zxf mysql-connector-java-5.1.45.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar
cp mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar  /opt/cloudera-manager/cm-5.13.3/share/cmf/lib/

3、 initialization CM5 The database of ( Master node )

/opt/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p scm scm
# View the configuration file , Discovery information has been written ：
vim /data/cm-5.13.3/etc/cloudera-scm-server/db.properties

4、Agent To configure ( Master node )

vim /opt/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini
server_host=cdh-master
cloudera_mysql_connector_jar=/usr/share/java/mysql-connector-java.jar

5、 Sync Agent To other nodes ( Master node )

scp -r /opt/cloudera-manager/cm-5.13.3 [email protected]*:/opt/cloudera-manager

6、 establish cloudera-scm user ( All nodes )

useradd --system --home=/opt/cloudera-manager/cm-5.13.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

7、 Get ready Parcels( Master node )

mkdir -p /opt/cloudera/parcel-repo/
cp CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel /opt/cloudera/parcel-repo/
cp CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha
cp manifest.json /opt/cloudera/parcel-repo/

# All nodes 
chown -R  cloudera-scm:cloudera-scm /opt/cloudera/
chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
chown cloudera-scm:cloudera-scm /opt/cloudera-manager 

vim /etc/sysctl.conf
vm.swappiness = 10
sysctl –p

# All nodes execute 
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

# Power on execution , Add the above statement 
chmod +x /etc/rc.d/rc.local
vim /etc/rc.local

9、 Start related scripts

/opt/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server start    ( Master node )
/opt/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent start     ( All nodes )
# It can also be used stop, restart

3、 ... and 、CDH5 Installation configuration

1. Sign in

http://172.21.161.32:7180/cmf/login, The username and password are admin/admin