In our current scenario, we have 4 Node cluster where one is master node (HDFS Name node and YARN resource manager) and other three are slave nodes (HDFS data node and YARN Node manager)
In this cluster, we have implemented Kerberos, which makes this cluster more secure.
Kerberos services are already running in the different server which would be treated as KDC server.
In all of the nodes, we have to do a client configuration for Kerberos which I have already written in my previous blog. please go through below kerberos authentication links for more info.
To start the installation of Hadoop HDFS and Yarn follow the below steps:
Prerequisites:
- All nodes should have an IP address as mentioned below
- Master : 10.0.0.70
- Slave 1 : 10.0.0.105
- Slave 2 : 10.0.0.85
- Slave 3 : 10.0.0.122
- SSH password less should be there from master node to all the slave node in order to avoid password prompt
- each node should communicate with each other.
- 1.8 OpenJDK should be installed on all four nodes.
- jsvc package should be installed on all four nodes.
- Need to make host file entry in all of the nodes to communicate with each other by name as local DNS.
- I am assuming Kerberos packages already installed in all of the four nodes and configuration has also done.
Lets begin the configuration:
add master node’s ssh public key in all of the worker’s nodes authorized_keys which would be found in ~/.ssh/authorized_keys
after adding keys, master node will be able to login in to all of the worker’s nodes without password or keys.
install jdk 1.8 in all four nodes
sudo apt-get install openjdk-8-jdk -yinstall jsvc too in all of the four nodes
sudo apt-get install jsvc -yplease make host file entry as mentioned in all of the four nodes
open /etc/hosts file
vim /etc/hostsadd below mentioned parameters for each node and also add one entry for Kerberos(10.0.0.33 EDGE.HADOOP.COM)
10.0.0.70 master
10.0.0.105 worker1
10.0.0.85 worker2
10.0.0.122 worker3
10.0.0.33 EDGE.HADOOP.COMI am performing here all the operation from root user.
master node:
Download Hadoop 3.0.0 from the official link of apache then extract it and move as hadoop directory.
cd /
wget https://archive.apache.org/dist/hadoop/core/hadoop-3.0.0/hadoop-3.0.0.tar.gz
tar -xzf hadoop-3.0.0.tar.gz
mv -v hadoop-3.0.0 hadoopnow we have to add environment variable in master node
for this, please add all the variables in .bashrc of user root
vim ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/hadoop
export PATH=$HADOOP_HOME/bin:$PATH
export HADOOP_CONF_DIR=/hadoop/etc/hadoop
export HADOOP_USER_NAME=root
export LD_LIBRARY_PATH=/hadoop/lib/native:$LD_LIBRARY_PATH
export HDFS_SECONDARYNAMENODE_USER=rootto reflect changes instantly please run source ~/.bashrc
please go on the configuration path of hadoop
cd $HADOOP_CONF_DIRHDFS configurations:
and Open core-site.xml and add these configurations parameter.
vim core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value> <!-- A value of "simple" would disable security. -->
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
</configuration>now open hdfs-site.xml and place below mentioned configuration
vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/name-node</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/data/data-node</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-hdfs/cache/ubuntu</value>
</property>
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>dfs.namenode.servicerpc-bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
<description>Whether clients should use datanode hostnames when connecting to datanodes
</description>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>true</value>
<description>Whether datanodes should use datanode hostnames when connecting to other d
</description>
</property>
<!-- General HDFS security config -->
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<!-- NameNode security config -->
<property>
<name>dfs.namenode.keytab.file</name>
<value>/home/ubuntu/keytabs/hdfs-master.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/_HOST@HADOOP.COM</value>
</property>
<property> <name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@HADOOP.COM</value>
</property>
<!-- Secondary NameNode security config -->
<property> <name>dfs.secondary.namenode.keytab.file</name>
<value>/home/ubuntu/keytabs/hdfs-master.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property> <name>dfs.secondary.namenode.kerberos.principal</name>
<value>hdfs/_HOST@HADOOP.COM</value>
</property>
<property> <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@HADOOP.COM</value>
</property>
<!-- DataNode security config -->
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:1004</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:1006</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>/home/ubuntu/keytabs/worker.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>root@HADOOP.COM</value>
</property>
<!-- Web Authentication config -->
<property> <name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/_HOST@HADOOP.COM</value>
</property>
</configuration>
open worker file and add all workers nodes DNS Name
vim workers
worker1
worker2
worker3Open the hadoop-env.sh file and replace with mentioned values
vim hadoop-env.sh
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/jre"
export HADOOP_HOME="/hadoop"
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
export HDFS_DATANODE_SECURE_USER=root
export HDFS_NAMENODE_SECURE_USER=root
YARN configurations:
create a yarn user in all of the nodes
useradd yarnadd mentioned parameters in container-executor.cfg
vim container-executor.conf
yarn.nodemanager.linux-container-executor.group=yarn
allowed.system.users=root,ubuntu,knoldus,Administrator,yarn
feature.tc.enabled=0
then give mentioned permission to container-executor.conf file
chmod 644 container-executor.confopen yarn-site.xml conf file and add mentioned configurations
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.dispatcher.exit-on-error</name>
<value>true</value>
</property>
<property>
<description>List of directories to store localized files in.
</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop-yarn/cache/ubuntu/nm-local-dir</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>/var/log/hadoop-yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
/hadoop/*,/hadoop/lib/*,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<name>1536</name>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4608</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4608</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>3072</value>
</property>
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx2457m</value>
</property>
<property>
<name>yarn.webapp.ui2.enable</name>
<value>true</value>
</property>
<!-- resource manager secure configuration info -->
<property>
<name>yarn.resourcemanager.principal</name>
<value>yarn/_HOST@HADOOP.COM</value>
</property>
<property>
<name>yarn.resourcemanager.keytab</name>
<value>/home/ubuntu/keytabs/yarn-worker.keytab</value>
</property>
<!-- remember the principal for the node manager is the principal for the host this yarn-site.xml file is on -->
<!-- these (next four) need only be set on node manager nodes -->
<property>
<name>yarn.nodemanager.principal</name>
<value>yarn/_HOST@HADOOP.COM</value>
</property>
<property>
<name>yarn.nodemanager.keytab</name>
<value>/home/ubuntu/keytabs/yarn-worker.keytab</value>
</property>
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>yarn</value>
</property>
</configuration>
now give desirable permissions to container-executor in all of the nodes
cd /hadoop/bin/
chown root:yarn container-executor
chmod 050 container-executor
chmod u+s container-executor
chmod g+s container-executorduplicate the Hadoop directory with all the configuration file(core-site.xml,hdfs-site.xml, and yarn-site.xml to all of 3 workers’ nodes
scp -r /hadoop worker1:/
scp -r /hadoop worker2:/
scp -r /hadoop worker3:/Please add below mentioned fields in environment variables of master node
vim ~/.bashrc
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=roothdfs and yarn configuration has been done.
Spark configuration and integration with YARN
download the spark binary from the mentioned path then extract it and move it as spark directory.
wget https://mirrors.estointernet.in/apache/spark/spark-2.4.6/spark-2.4.6-bin-without-hadoop-scala-2.12.tgz
tar -xvf spark-2.4.6-bin-without-hadoop-scala-2.12.tgz
mv -v spark-2.4.6-bin-without-hadoop-scala-2.12 sparkSet the environment variable for spark
vim ~/.bashrc
export SPARK_HOME=/spark
export PATH=$SPARK_HOME/bin:$PATH
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$SPARK_HOME/sbin
Now rename spark default template
mv $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.confEdit $SPARK_HOME/conf/spark-defaults.conf and add mentioned parameters.
# Example:
spark.master yarn
# spark.eventLog.enabled true
# spark.eventLog.dir hdfs://namenode:8021/directory
# spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 512m
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.yarn.am.memory 512m
spark.executor.memory 512m
spark.driver.memoryOverhead 512
spark.executor.memoryOverhead 512
rename and open slaves file under $SPARK_HOME/conf and paste below mentioned workers
mv -v slaves.template slaves
vim slaves
worker1
worker2
worker3mv spark-env.sh.template spark-env.shadd below parameter on spark-env.sh
export SPARK_DIST_CLASSPATH=$(hadoop --config $HADOOP_CONF_DIR classpath)Now spark and yarn integration has been done.
it’s time to start the services of hdfs and yarn. before starting the configuration first need to format namenode.
hdfs namenode -formatNow start the services of hdfs
cd /hadoop/sbin
./start-dfs.shThis will start name node in master node as well as data node in all of the workers nodes
We can check and verify by running jps in all of the nodes
jps
15126 Jps
18823 NameNode
19160 SecondaryNameNodestart the service of yarn
cd ~/hadoop/sbin/
./start-yarn.shthis will start resource manager service in master node and node manager services in all of the worker nodes.
For accessing the url of hdfs. please use below link
For accessing the url of YARN
Note: please create the principal of Kerberos and keytab files as per your desire. because these would be used in the configuration file of hdfs and yarn.
Conclusion:
Here, we have created four-node HDFS, YARN clusters with one master node, and 3 worker nodes, and spark job will be running on top of YARN. Kerberos is also implemented for secure communication.
References:
1 thought on “Install/Configure Hadoop HDFS,YARN Cluster and integrate Spark with it11 min read”
Comments are closed.