1, Prepare resources (1 ~ 4 downloads are free resources, or you can download them yourself)

1. Operating system (CentOS7)
   (1) desktop version is used as master:
   Baidu online disk link: Click download
   Extraction code: wz4z
   (2) the version without desktop is used as a slave:
   Baidu online disk link: Click download
   Extraction code: gjyf
2. Hadoop-2.9.2
   Baidu online disk link: Click download
   Extraction code: gtg5
3. jdk1.8
   Baidu online disk link: Click download
   Extraction code: 8vop
4. FileZllia
   Baidu online disk link: Click download
   Extraction code: 5oqu
5. VMWare
   I can't help you. Find your own resources!

2, Cluster overview:

A total of 3 virtual machines are configured. The 3 virtual machines are named master, slave01 and slave02 respectively;
The user of each virtual machine uses the root user with the password of 123456;

3, Preparatory work

Please install three virtual machines (one with desktop and the other two without desktop) in VMware, modify the hostname and hosts files, and the hosts file should contain the mapping relationship between the three virtual machines and the corresponding ip address. For specific configuration, please refer to Previous blogs.

4, Password free login

Each virtual machine performs the following operations:
    1. Modify sshd_config configuration file:

vi /etc/ssh/sshd_config

    2. Modify line 43: delete the comment of PubkeyAuthentication yes;
    3. On line 44, add:

RSAAuthentication yes

    4. Restart sshd Service:

systemctl restart sshd.service

    5. Generate key:

# The first command encountered a direct enter
ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

    6. Password free other virtual machines:

ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01
ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02

    7. Secret free:

ssh master

5, Configure master

1. Configure HADOOP_HOME environment variable:
   (1) create Hadoop SH script:

vi /etc/profile.d/hadoop.sh

(2) to Hadoop The SH script writes the following:

export HADOOP_HOME=/usr/local/hadoop-2.9.2
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

(3) make Hadoop SH script takes effect:

source /etc/profile.d/hadoop.sh

1. Configure $HADOOP_HOME/etc/hadoop/hadoop-env.sh profile:
   (1) amend act 25:

export JAVA_HOME=/usr/java/jdk1.8.0_311

(2) add line 26 to read:

export HADOOP_SSH_OPTS='-o StrictHostKeyChecking=no'

(3) amend line 113 to read:

export HADOOP_PID_DIR=${HADOOP_HOME}/pids

1. Configure $HADOOP_HOME/etc/hadoop/mapred-env.sh profile:
   (1) amend line 16 to read:

export JAVA_HOME=/usr/java/jdk1.8.0_311

(2) amend line 28 to read:

export HADOOP_MAPRED_PID_DIR=${HADOOP_HOME}/pids

1. Configure $HADOOP_HOME/etc/hadoop/yarn-env.sh profile:
   (1) amend line 23 to read:

export JAVA_HOME=/usr/java/jdk1.8.0_311

(2) add the following line at the end of the document:

export YARN_PID_DIR=${HADOOP_HOME}/pids

1. Configure $HADOOP_HOME/etc/hadoop/core-site.xml configuration file:
   Write the following in < Properties > < / Properties >:

		<property>
                <name>fs.defaultFS</name>
                <value>hdfs://192.168.159.3:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/local/hadoop-2.9.2/hdfsdata</value>
        </property>

1. Configure $HADOOP_HOME/etc/hadoop/mapred-site.xml configuration file:
   Write the following in < Properties > < / Properties >:

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

1. Configure $HADOOP_HOME/etc/hadoop/yarn-site.xml configuration file:
   Write the following in < Properties > < / Properties >:

        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>master</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>

1. Modify $Hadoop_ The home / etc / Hadoop / slaves file is:

slave01
slave02

6, Configure slave

Transfer the configuration file in the master to the slave:

scp -R /ect/profile.d/hadoop.sh slave01:/ect/profile.d
scp -R /ect/profile.d/hadoop.sh slave02:/ect/profile.d
scp -R /usr/local/hadoop-2.9.2 slave01:/usr/local
scp -R /usr/local/hadoop-2.9.2 slave02:/usr/local

Restart the slave:

reboot

7, Start Hadoop

You can only run the following commands on the master:

hdfs namenode -format
start-dfs.sh
start-yarn.sh
mr-jobhistory.sh start historyserver

8, Close Hadoop

You can only run the following commands on the master:

mr-jobhistory.sh stop historyserver
stop-yarn.sh
stop-dfs.sh