Integrating LVM with Hadoop

đź”…Integrate LVM with Hadoop and
provide Elasticity to DataNode Storage
đź”…Increase or Decrease the Size of Static
Partition in Linux.
đź”…Automating LVM Partition using Python-Script.

Step -1: Launching 2 instances one for Hadoop master(Name node) and the other for Hadoop slave(Data node).

Step-2: Download and install JDK n Hadoop software in both the nodes:



To install we will use the following commands:

yum install jdk-8u171-linux-x64.rpm -y

rpm -ivh hadoop-1.2.1–1.x86_64.rpm — force -y

Step-3: Configuring name node n data node by updating hdfs-site.xml and core-site.xml

Name node hdfs-site.xml file

Name node core-site.xml file(IP: n port 9001)

data node hdfs-site.xml

data node core_site.xml(ip: master IP(public IP of name node instance) n port 9001)

Step-4: Format the namenode

Step-5: Create 2 EBS volumes and attach them to slave (data node)

Step-6: Create Physical volumes, logical volume, volume group for these 2 volumes so that we can use them.

If pvcreate cmd is not available then download it

Creating physical volumes

Create Volume group:

Creating Logical Volume:

Step-7: Format the partition

Step-8: Mount the partition to data node folder

Step-9: Start the name node n data node services

Checking the size of the data node

Step- 10: Extending Logical Volume size to increase the data node space

Step-11: Now again checking the report to see if the size is increased or not

The size is now increased from 3Gib to 6Gib




Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Small API To Full-On Business Solution In Less Than A Year

How to maintain different branches in Git for your project

Exception Handling Card

Three Lines of Code

Home Lab Guides: Proxmox 6 — IPFire Firewall

Maximum Product Subarray

Maximum Sum Subarray

Clean Code Development | Integration Operation Separation Principle

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aakash Bhardwaj

Aakash Bhardwaj

More from Medium

7 Major Benefits Of Data Lakes

Getting started with streaming

Apache Druid — Druid SQL in Java

Cloud and Big Data: Top Trends