Integrating LVM With Hadoop To Provide Elastic Storage Kind Of Feature To DataNode

Hello Guyyyyssss 😃😃

In this Article , I will gonna to show you how we can integrate LVM with Hadoop to provide Elastic Storage kind of feature to the datanode..(Why Datanode ???.. Because , Datanodes are the one where the data is stored in the hadoop cluster..At the master side only the metadata is stored…That’s why only datanode 😃)

I am considering that u guys are familiar with the hadoop cluster and know how to set-up your own hadoop cluster..So, here we 🚀

Currently, In my cluster I have one master node and one slave node…

I attached hard disk(in my case VDI since I am using Virtual Machine) of size 1GiB to my operating system..Then

1. I created it’s partition using fdisk command utility and divided it into the two parts 500MB + 500MB
2. Then , I created the pv from both the partition using command pvcreate <name of the partition>
3. Then , I created vg from both the partition using the command vgcreate <vg-name-we-want-to-give> <name-of-partition-1> <name-of-partition-2>

Then , I created the LVM(of size 500 MiB) from the vg of approx size 1024 MB

Now ,I formatted the partition and then mount it to the /slave directory(directory of the data node) and then start the data node service

Now , If I see the output of hadoop dfsadmin -report then it looks like

Now , I stored a test file in the data lake provided by the hadoop cluster

Now , I increased the size of the lvm of the data node from 500M to 700M(approx)

But ,the output of the df -hT command looks like

Because , the increased portion has not been formatted yet..So , I made the use of resize2fs command for doing the online formatting..(resize2fs only supports online extending not shrinking)

Now ,the df -hT command’s output looks like

Now , I check the output of hadoop dfsadmin -report command

Now , I tried to reduce the size of the lvm of the storage of data node again using the resize2fs command

So, from here we can see that resize2fs command only supports on-line expanding not shrinking..So , for this first we have to make the folder offline by stooping the data node services and also unmount the lvm from the /slave directory

Then , I just mount the lvm to the /slave directory and again start the datanode service

Then , I checked the output of hadoop dfsadmin -report

At the end I also checked whether my data(here /text file), is there in the data lake or get lost in this process of resizing..And what I found is , it was there in the data lake 😃

Thanks For Reading..!!! 🔥

Hope, I was able to give u atleast one unknown information in this article.

See u in the next one ..

Signing Off … 😃