1. Preparations

1.1 Off-line deployment is mainly divided into three parts

MySQL offline deployment
CM Offline Deployment
Parcel File Offline Source Deployment

1.2 Planning

1.3 Download Source

• CM
  cloudera-manager-centos7-cm5.16.1x8664.tar.gz
• Parcel
  CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
  CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
  manifest.json
• JDK
  jdk-8u202-linux-x64.tar.gz
• MySQL
  mysql-5.7.26-el7-x86_64.tar.gz
• MySQL jdbc jar
  mysql-connector-java-5.1.47.jar
  After downloading, rename and delete the version number.
  mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar

2. Cluster Node Initialization

2.1 Ajiyun purchases three ECS hosts and pays for them by volume

CentOS 7.2 Operating System, 2 Core 8G Minimum Configuration

2.2 Windows Configuration hosts File

Window: C:\windows\system32\drivers\etc\hosts

alternate public address
47.110.15.86		hadoop001
47.110.10.192		hadoop002
47.110.14.197		hadoop003

2.3 Linux configuration hosts file

Intranet address
echo "172.16.189.132		hadoop001" >>/etc/hosts
echo "172.16.189.133		hadoop002" >>/etc/hosts
echo "172.16.189.131		hadoop003" >>/etc/hosts

2.4 Close firewalls and empty rules for all nodes

systemctl stop firewalld
systemctl disable firewalld
iptables -F

2.5 Close selinux for all nodes

vi /etc/selinux/config
 Change SELINUX = en forcing to SELINUX=disabled
 Restart is required after setup to take effect

2.6 Setting Clock Synchronization for All Nodes

The mainframe on the Ali Cloud is clock synchronized. I didn't configure it. I skipped this step.

6.1.time zone
[root@hadoop001 ~]# date
Sat May 11 10:07:53 CST 2019

[root@hadoop001 ~]# timedatectl
Local time: Sat 2019-05-11 10:10:31 CST
Universal time: Sat 2019-05-11 02:10:31 UTC
RTC time: Sat 2019-05-11 10:10:29
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: yes
DST active: n/a

#View command help
[root@hadoop001 ~]# timedatectl --help
timedatectl [OPTIONS...] COMMAND ...
Query or change system time and date settings.
-h --help Show this help message
--version Show package version
--no-pager Do not pipe output into a pager
--no-ask-password Do not prompt for password
-H --host=[USER@]HOST Operate on remote host
-M --machine=CONTAINER Operate on local container
--adjust-system-clock Adjust system clock when changing local RTC mode
Commands:
status Show current time settings
set-time TIME Set system time
set-timezone ZONE Set system time zone
list-timezones Show known time zones
set-local-rtc BOOL Control whether RTC is in local time
set-ntp BOOL Control whether NTP is enabled

#See which time zones
[root@hadoop001 ~]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako

#All nodes set asia-Shanghai time zone
[root@hadoop001 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop002 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop003 ~]# timedatectl set-timezone Asia/Shanghai

6.2.time
#All nodes install ntp
[root@hadoop001 ~]# yum install -y ntp

#Choose Hadoop 001 as the main node of ntp
[root@hadoop001 ~]# vi /etc/ntp.conf
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#When external time is not available, local hardware time can be used.
server 127.127.1.0 iburst local clock
#Which machine segments are allowed to synchronize the time
restrict 172.19.7.0 mask 255.255.255.0 nomodify notrap

#Open ntpd and view status
[root@hadoop001 ~]# systemctl start ntpd
[root@hadoop001 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: d
isabled)
Active: active (running) since Sat 2019-05-11 10:15:00 CST; 11min ago
Main PID: 18518 (ntpd)
CGroup: /system.slice/ntpd.service
??18518 /usr/sbin/ntpd -u ntp:ntp -g
May 11 10:15:00 hadoop001 systemd[1]: Starting Network Time Service...
May 11 10:15:00 hadoop001 ntpd[18518]: proto: precision = 0.088 usec
May 11 10:15:00 hadoop001 ntpd[18518]: 0.0.0.0 c01d 0d kern kernel time sync enabl
ed
May 11 10:15:00 hadoop001 systemd[1]: Started Network Time Service.

#Verification
[root@hadoop001 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) .LOCL. 10 l 726 64 0 0.000 0.000 0.000

#Other slave nodes stop and stop using ntpd services
[root@hadoop002 ~]# systemctl stop ntpd
[root@hadoop002 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.

[root@hadoop002 ~]# /usr/sbin/ntpdate hadoop001
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec

#Synchronized Hadoop 001 node time every morning
[root@hadoop002 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001

[root@hadoop003 ~]# systemctl stop ntpd
[root@hadoop004 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.

[root@hadoop005 ~]# /usr/sbin/ntpdate hadoop001
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec

#Synchronized Hadoop 001 node time every morning
[root@hadoop003 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001

2.7 JDK for deploying clusters

mkdir /usr/java
tar -xzvf jdk-8u45-linux-x64.tar.gz -C /usr/java/
#Keep in mind that the users and groups to which they belong must be corrected
chown -R root:root /usr/java/jdk1.8.0_45
echo "export JAVA_HOME=/usr/java/jdk1.8.0_45" >> /etc/profile
echo "export PATH=\${JAVA_HOME}/bin:\${PATH}" >> /etc/profile  #Before \ Transliteration
source /etc/profile
which java

2.8 Hadoop 001 node deploys MySQL 5.7 offline

Blog Connection
https://blog.csdn.net/greenplum_xiaofan/article/details/97670427

2.9 Create metadata base and user of CDH, database and user of amon service

create database cmf DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'Ruozedata123456!';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'Ruozedata123456!';
flush privileges;

2.10 Hadoop 001 node deployment mysql jdbc jar

mkdir -p /usr/share/java/
cp mysql-connector-java.jar /usr/share/java/

3. CDH deployment

3.1 Offline deployment of cm server and agent

1.1.Create directories and unzip all nodes
mkdir /opt/cloudera-manager
tar -zxvf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager/

1.2.All node modifications agent Configuration, pointing server Nodes hadoop001
sed -i "s/server_host=localhost/server_host=hadoop001/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.3.Main Node Modification server Configuration:
vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop001
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=Ruozedata123456!
com.cloudera.cmf.db.setupType=EXTERNAL

1.4.All nodes create users
useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1.5.Modify Users and User Groups
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

3.2 Hadoop 001 node deploys offline parcel source

2.1.Deployment offline parcel source
$ mkdir -p /opt/cloudera/parcel-repo
$ ll
total 3081664
-rw-r--r-- 1 root root 2127506677 May 9 18:04 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root 41 May 9 18:03 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 841524318 May 9 18:03 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 185515842 Aug 10 2017 jdk-8u144-linux-x64.tar.gz
-rw-r--r-- 1 root root 66538 May 9 18:03 manifest.json
-rw-r--r-- 1 root root 989495 May 25 2017 mysql-connector-java.jar

$ cp CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo/
#Keep in mind that when cp is being renamed, remove 1, or CM will continue to download if the file download is incomplete during the deployment process.
$ cp CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
$ cp manifest.json /opt/cloudera/parcel-repo/

2.2.Directory Modification Users and User Groups
$ chown -R cloudera-scm:cloudera-scm /opt/cloudera/

3.3 All nodes create software installation directory, user and user group permissions

mkdir -p /opt/cloudera/parcels 
chown -R cloudera-scm:cloudera-scm /opt/cloudera/

3.4 hadoop001 node starts Server

4.1. Start the server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

4.2. A_Li cloud web interface, set the Hadoop 001 node firewall to open 7180 ports
 4.3. Wait for 1 minute and open the password of http://hadoop001:7180: admin/admin
 4.4. If you can't type it properly, go to the server's log and check the errors carefully according to the errors.

3.5 All nodes start Agent

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

3.6 Next, all Web interface operations

http://hadoop01:7180
 Account password: admin/admin

3.7 You are welcome to use Cloudera Manager - Terms and Conditions of End User License. Checklist

3.8 Welcome to Cloudera Manager - which version do you want to deploy? Choose Cloudera Express Free Version

3.9 Thank you for choosing Cloudera Manager and CDH

3.10 is the guidance host for CDH cluster installation. Select [current management host], check all

3.11 Selection Repository

3.12 Cluster Installation - Selected Parcel is being installed if

If the local parcel offline source is configured correctly, the "download" phase will be completed in an instant, and the remaining stages will be determined by the number of nodes and the internal file network.

3.13 Check Host Correctness

I saved time in doing this. What I did was take temporary effect.
When you're done, BACK the last step and come back.

[root@hadoop001 parcel-repo]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop001 parcel-repo]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

13.1. It is recommended that / proc / sys / VM / swappines be set to a maximum value of 10.
Swappines value controls the active attempt of the operating system to swap memory;
swappiness=0: It means to maximize the use of physical memory, followed by swap space.
swappiness=100: means actively partitioning with swap and moving data from memory to swap space in time;
If it's a mixed server, it's not recommended to ban swap altogether. You can try to reduce swapping.
Temporary adjustment:
sysctl vm.swappiness=10
 Permanent adjustment:
cat << EOF >> /etc/sysctl.conf
# Adjust swappiness value
vm.swappiness=10
EOF

13.2. Compression of transparent large pages may cause heavy performance problems. It is recommended that this setting be banned.
Temporary adjustment:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
 Permanent adjustment:
cat << EOF >> /etc/rc.d/rc.local
# Disable transparent_hugepage
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
 # centos7.x system, you need to grant line-executing permissions to'/ etc/rc.d/rc.local'files
chmod +x /etc/rc.d/rc.local

3.14 Custom Services, Choose to Deploy Zookeeper, HDFS, Yarn Services

3.15 Custom Role Assignment

3.16 Database Settings

3.17 Revise the settings by default

3.18 First shipment

3.19 Congratulations!

3.20 Home Page