Configure Hadoop cluster using ansible playbooks
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyPIQ321shnXqBOt2MsvAPXUHs6_brpDExfexZGBMWVraEgTrL7BPDiOJF_dy7ouQLus4ffOfAcHsuC4It4JVMslnQnL2u8G2rgBUt0EdsZkpCWfS36qHoemyoOwv-jdX84rtogKqAXUw/s16000/1_GLjbQ3SfYO792G0oThm_2g+%25281%2529.png)
Hadoop is open-source software for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power, and the ability to handle virtually limitless concurrent tasks or jobs.
RED HAT ANSIBLE
Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
![](https://miro.medium.com/max/320/0*4w7ZW_8GlteOgjEH.png)
Ansible is Designed for multi-tier deployments, Ansible models your IT infrastructure by describing how all of your systems inter-relate, rather than just managing one system at a time.
Task Objective :
Configure Hadoop and start cluster services using Ansible Playbook.
Configuration plan
step 1: Connect to the managed node
step 2: download JDK and Hadoop
step 3: install JDK and Hadoop
step 4: update core-site.xml and hdfs-site.xml
step 5: Format Namenode
step 6: start services
To get a true feeling of automation please use playbook only do not use any command in between
NOTE: all the source code is in GitHub linked provide in the last
Let’s Get Started
To install ansible run the following command:
pip3 install ansible
you need to create an ansible directory in
etc/ansible/ // ansible directory
Inventory File[Data.txt]
Ansible uses this file to check host connectivity it is a kind of IP database.
[MasterNode]192.168.0.112 ansible_connection=ssh ansible_user=root ansible_ssh_pass=root[SlaveNode]192.168.0.113 ansible_connection=ssh ansible_user=root ansible_ssh_pass=root
Configuration File[ansible.cfg]
Ansible uses this file to check its configurations.
it is located at /etc/ansible/ansibel.cfg// you need to create file[defaults]inventory = /data.txthost_key_checking = Falsedeprecation_warnings = False
step 1: Connect to the managed node
to check the connectivity of the managed node run the following command
ansible all -m ping
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB9T3Z1ACwN8YFs04cX8mAVELv8epEj7Bjx8N5oU5hh_0WUWHUTc-dCGcJ4KqkcG5_W273If6nmPHELE7M-4c4O3OCIbaKhyphenhyphenPQyPHrNa-Y7uuztFPhdvEVl_btc4qugnhr6COLKfw0_gU/w640-h320/1_xbO3lIpvB6GPbcm9QpkBiA.png)
NOTE: I have already downloaded JDK and Hadoop using ansible-playbook.
Download From Here:
JDK:https://github.com/frekele/oracle-java/releases/download/8u171-b11/jdk-8u171-linux-x64.rpmHadoop:https://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1-1.x86_64.rpm
To run Playbook use the following command
ansible-playbook FileName.yml
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFo2W71Q3tyqbC61_3XppdmcbAu8iYtOz4d5_FLCYdHOhYGRe7g2BieBexobXKAaa61_qgSAmlfJcdfavHoWEi6d6pQ2m6RAOPRxdSeVhysewJod35FQIRaLaFfEHmPiolrjGkHosufow/w640-h320/1_k3E_rihgHnaXg4fbEQk-0g.png)
To transfer files between Linux system run the following command
scp -r file1 file2 'ip of destination:/path to save file'
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXQ1b552aq3bxOVCXDukkW1SCpBupF1vI6a1PQ_wfPnCd13Cv2VHVjZnYFHQLxKspi-QFZ6wxZnUmZsJNsfcUwEElGQQDj_l3Q7RbQHehSaiRtcR4TV8yJXuadp7gRBLbEmAVGKDHzHYY/w640-h320/1_lxRd4GzSKCyTl0_MQRxsXw.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqMThWDPnrOwxvMdq-qjevhoeGU_DFijYryhE2EA7hqY_wxhpNVb5u9zkWkPRfBufeuhVwCJYh1GYC_qNkYVDo9Eha7lFXKPnW0NB_GlVQtE3TM3kLj25H9lCrnReoTq894utAoptxeTo/w640-h260/1_tNivKRnS30HmPpU4LpfJKQ.png)
step 3: install JDK and Hadoop
to install JDK and Hadoop we have an ansible playbook you will get in my GitHub
we will use the shell module from the ansible because JDK and Hadoop is not part of yum repository
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlBzjQnXEb94oWxu4x8bq0khqKgZrCUZaOL7VydSBqyHWLNbLmduWXHrKwHI2878iBk9YOVWk7bcLW9J6sf1xplqsmUnAxdRVV2kMmi9vUrccbDx220_x23XtCNzO-CWqf2IjOe3w90sg/w640-h320/1_fPrx0h3a7AlMDvZbHGp4Zg.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBQL6m0wXdilx1oettYS23dNazfqQXLRjx3xp6jVt9GnxIazXtiqBCQNhXGsrzllh00eGf0ngTZkZaERR1Sg2LA76Y7Efnq7P4LSAbs4UoYZkvoJP5VdbSmQgTFv-vxcnhuOBvYtbku_s/w640-h320/1_J129UT7uE6-ObuToaGsj3w.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivQoHoIHNl47ijaqUdgqaQr3D84RIE3AY8Lw6QQaWVfSMwsHRp8a_vBZx7IFgMLqmnu381zvoZo_3nz-dmwD_XgU4yxlZh0DVBjvdNhpPPLNqJxXtFB00Et_4Qi18mLnSOo24yYZUqFGE/w640-h320/1_Y2CKzFWFh5S-m7vIAGm0IA.png)
NOTE: only Hdfs-site.xml will be different in master and data node. Core-site.xml will remain the same in both.
hdfs-site.xml // will be different in both // Master Node<configuration><property><name>dfs.name.dir</name> // telling take following directory<value>/master</value> //Folder path</property></configuration>
core-site.xml // will be same in both
<configuration><property><name>fs.default.name</name><value>hdfs://192.168.0.112:9001</value> // any port you can give</property></configuration>
NameNode or MasterNode
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijj6g7wrU_cVi89lwSTg6QL05tv-r16U5MKcFebg-644YtVf9y2y8IKcz9OezFTMcCQMr0Rc_D4tRfnEH-_oEsYNW_GMogekNMVUUh6KhzKx0wiJZMKg6fdNkteUVZG8zFjR_q13PEsJE/w640-h320/1_9Q0oYGMfoFGGTPXo38bSvg.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLalnmm8lvdpqeouGUViFkOWeOtT3iA5OJVzuA63Lv51OvfVBRst5hKkwTTqKhudWLQarxddJO1CdHZzUffmYlcsZ-0lkRqz-oDA1NJSeSVruNPGhSeIHeiqqyXV7GOtdGNERKCpAME8E/w640-h320/1_daRZVHtLMiZPrUqcnQk1Vg.png)
hdfs-site.xml // will be different in both // slave Node<configuration><property><name>dfs.data.dir</name> // telling take following directory<value>/slave</value> //Folder path</property></configuration>
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje4_d7gJIDoFm0b4zAIZ_GdRSzAfZwMDv8UA19EcchX7bs7q3tbdZirIFV13aFSlQlEHz_DsQIlrcioPMJWhRx6VE7UhOjlN153xTA_muQMnVd1ZdcZISQ4SBXodYrzwPrtCSEdfnEqBg/w640-h320/1__E2rdqnWZVqR8M8eSXJ7hA.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA1Q84AOjfREdVCn8-ZQ_Vi5srzNS4YTXVCsXzISK3Q9YCtxcHs3ZhgdHUyEC-72nCVnz6P2g6-J8RLL-xMxBUesyL45DFhqFcrq-Ij-p6WUu9gwA2zrJI2_raPVi9BEQd77mDXD7e3F0/w640-h320/1_xlMOpq9cCnXGjuw2BHM56g.png)
Hadoop uses its own file system that is HDFS(Hadoop file system) so we need to format the MasterNode only no need to format slave node Because management is handled by Master Node
we have the following command to do this but we will use ansible Playbook for this as well
hadoop namenode -format
We need to pass the command
echo Y | hadoop namenode -format
in the shell module because this command takes user input while formatting name node
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaxbCiaJfWF-rkgDWOn7OrBUcGP5NAseQVgezuY4JtROQGAjkQq2l-Zw97EohGwILm_eProzylMgAMosQS-Fu1UR_3J-eJSkMXUPuaHXOOaWIUMUmMvIOePKu-YMEakj_3rCX-zlqe2GY/w640-h320/1_LfliZD7vBYohhIVVvOYGlA.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKyRkAgwFjPWWCBhrUdDvDQkHe7v9Pim1knKgWI98nLhgr3LufybswNtkev2zrI61KskJqquZhyY801dA_W-u70eIyx5U8XkNvGEOHjiORl_8j_5CMzUWJ_qyDFHZZFWkx00NfF0IDAoo/w640-h320/1_mHfit_I1ntiQ0jqqaG-aHw.png)
Note: MasterNode already started so we will see DataNode
To start the Cluster services we have command but we will use ansible-playbook only
Master
hadoop-daemon.sh start namenode
// use 'stop' insted of 'start' to stop servicesSlave
hadoop-daemon.sh start datanode
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzz3ANk9_BuxhM9kGd_Q_Xs9tcs-70w3QVoRStk7JKqeQi9RDlr9iLGR9K2kLL3qWfmOXW-WxpYZGjSD9j6QGVtX2C0CX-JerOHRY44Mu3Z2wmNyep202tNg6680V2jVW2zTQB3zE5gEw/w640-h320/1_88Es_9dGBCTS_fF8JYv-lg.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_P51KGd1bYaEPzlZFJ4cdfDO82XSdWiWYW4-tsBpFr2nHWrQHzGRhhIxSvigsU0PLD-WxwppEb0eOzCfRjfSxYlKRYxshKZkad7Rl2LXMtYAYZUPi8WpZwMCQXXzQ4u5VR5idWPz8sxQ/w640-h320/1_p7x4HnX5FwJxqMKiGFTvsA.png)
to access GUI use the following URL in the browser
http://'MasterNode IP':50070 // 50070 GUI port of hadoop
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBcmZZqZFAuOh01V57nDhPwkV2VMzZm-tN7ntl6BMvVew_dDj3UepHb9UcHvWrzQk3CBRdaFnKaF-vOchuvhojJZMaN9yGtaU7WoiZnhWBpb88xHKsu2mz2Q1o5aXu6AYtyBVVnTB5V1E/w640-h320/1_5zxWZcoe4npce1AUjNULKA.png)
![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu73lXQvCdQjjEF7JGpFmYn6rRFhoH-vZSLxlbAsCyjOtGJ1_3etRdGSX_Tzbo43B-JCDVCSMuq5SlaZAAxJMSUQSNMEcUkXdQUKeU3uTSpbd17COWoTEBHt2MqCXKYg1RmewY8wZK6kQ/w640-h320/1_8RPPcMzYexSc3KFCejli2Q.png)
GitHub Link:
https://github.com/venkateshpensalwar/ARTH/tree/main/Ansible/Configure%20Hadoop
Conclusion:
we have learned how to configure Softwares which have clustering technology. In this way, we can set up Hadoop clusters with the help of Ansible Automation.
Hope this blog is helpful to you!!!!
No comments