Hadoop 2.7.3 configures multiple namenodes (federation clusters) in the cluster

http://blog.csdn.net/wild46cat/article/details/53423472


Hadoop 2.7.3 configures multiple namenodes (federation clusters) in the cluster

First of all, configuring multiple namenodes in a cluster and using secondary Namenode in a cluster are two completely different things. I will write an official translation of haoop later, explaining the difference between the two. Let's talk about it briefly. The second aryNamenode's role is to share the pressure on the namenode and help the namenode do some processing regularly. Configuring multiple namenodes is equivalent to configuring a federated cluster, where each anmenode does not communicate and manages its own namespace.

Okay, here we go.
Of course, the premise of completing this configuration is:
1. Has been able to configure a hadoop cluster with a single namenode.
2. The haoop cluster should be fully distributed (pseudo-distributed is not tested, but single-point estimation is not feasible).

1. Hardware environment:
host1 192.168.1.221
host2 192.168.1.222
host3 192.168.1.223

Configuration files
Where host1 is used as a namenode, host2 as a namenode, and host3 as a datanode.
Configuration file (same on each host): hdfs-site.xml
  1. <?xml version="1.0" encoding="UTF-8"?>  
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
  3. <configuration>  
  4.   
  5.     <property>  
  6.         <name>dfs.namenode.name.dir</name>  
  7.         <value>file:/home/hadoop/dfs/name</value>  
  8.     </property>  
  9.     <property>  
  10.         <name>dfs.datanode.data.dir</name>  
  11.         <value>file:/home/hadoop/dfs/data</value>  
  12.     </property>  
  13.     <property>  
  14.         <name>dfs.replication</name>  
  15.         <value>2</value>  
  16.     </property>  
  17.     <property>  
  18.     <name>dfs.webhdfs.enabled</name>  
  19.     <value>true</value>  
  20.     </property>  
  21.     <property>  
  22.     <name>dfs.datanode.max.transfer.threads</name>  
  23.     <value>4096</value>  
  24.     </property>  
  25.   
  26.     <property>  
  27.         <name>dfs.federation.nameservices</name>  
  28.         <value>host1,host2</value>  
  29.     </property>  
  30.   
  31.     <property>  
  32.         <name>dfs.namenode.rpc-address.host1</name>  
  33.         <value>host1:9000</value>  
  34.     </property>  
  35.     <property>  
  36.         <name>dfs.namenode.http-address.host1</name>  
  37.         <value>host1:50070</value>  
  38.     </property>  
  39.     <property>  
  40.         <name>dfs.namenode.secondary.http-address.host1</name>  
  41.         <value>host1:9001</value>  
  42.     </property>  
  43.       
  44.     <property>  
  45.         <name>dfs.namenode.rpc-address.host2</name>  
  46.         <value>host2:9000</value>  
  47.     </property>  
  48.     <property>  
  49.         <name>dfs.namenode.http-address.host2</name>  
  50.         <value>host2:50070</value>  
  51.     </property>  
  52.     <property>  
  53.         <name>dfs.namenode.secondary.http-address.host2</name>  
  54.         <value>host2:9001</value>  
  55.     </property>  
  56. </configuration>  


Configuration file on host1: core-site.xml
  1. <?xml version="1.0" encoding="UTF-8"?>  
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
  3.   
  4. <configuration>  
  5. <property>  
  6.         <name>fs.defaultFS</name>  
  7.         <value>hdfs://host1:9000</value>  
  8.     </property>  
  9.     <property>  
  10.         <name>hadoop.tmp.dir</name>  
  11.         <value>file:/home/hadoop/tmp</value>  
  12.     </property>  
  13.     <property>  
  14.         <name>io.file.buffer.size</name>  
  15.         <value>131702</value>  
  16.     </property>  
  17. </configuration>  


Configuration file on host2: core-site.xml
  1. <?xml version="1.0" encoding="UTF-8"?>  
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
  3.   
  4. <configuration>  
  5. <property>  
  6.         <name>fs.defaultFS</name>  
  7.         <value>hdfs://host2:9000</value>  
  8.     </property>  
  9.     <property>  
  10.         <name>hadoop.tmp.dir</name>  
  11.         <value>file:/home/hadoop/tmp</value>  
  12.     </property>  
  13.     <property>  
  14.         <name>io.file.buffer.size</name>  
  15.         <value>131702</value>  
  16.     </property>  
  17. </configuration>  


Configuration file on host3: core-site.xml (df.defaultFS here is configurable to any one, as you need to explain)
  1. <?xml version="1.0" encoding="UTF-8"?>  
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
  3.   
  4. <configuration>  
  5. <property>  
  6.         <name>fs.defaultFS</name>  
  7.         <value>hdfs://host1:9000</value>  
  8.     </property>  
  9.     <property>  
  10.         <name>hadoop.tmp.dir</name>  
  11.         <value>file:/home/hadoop/tmp</value>  
  12.     </property>  
  13.     <property>  
  14.         <name>io.file.buffer.size</name>  
  15.         <value>131702</value>  
  16.     </property>  
  17. </configuration>  

Note that host3 as a datanode does not use the configuration in core-site.xml, while host1 and host2 read the local core-site.xml configuration file first when they read the file.
3. Test screenshots

Here is a simple test to show and prove that the two namenode s have their own namespaces separately.
First, start hadoop:


After startup, look at two namenode s from the web:


Create a directory in host1.

Folder in host1:


Create a directory in host2.

The folder in host2:


This shows that the two namenode s are separated and keep their corresponding tables of file blocks.

Keywords: xml Hadoop encoding

Added by Grunge on Tue, 21 May 2019 21:37:27 +0300