Contributing limited/specific amount of storage by datanode to a hadoop cluster using partitions

we have a hadoop cluster with one namenode and 4 datanodes with the capacities mentioned in the below image which is about 10GB, 10GB, 10GB and 8GB.

Now we want that the 4th datanode contribute only 5Gb storage (which is currently giving about 8Gb storage).

For this we can use the concept of partitions in linux. so, we can make a new partition of 5Gb (or any amount you want ) and then mount it to our folder which we are using for hadoop (in my case it is /dn4. we can find it in the hdfs-site.xml file as shown below)

Now lets make a new partition in our disk to which our dn4 folder is currently mounted. There can be many disks in our system and to find from which disk the folder is taking storage, we can use the command → df -h

So, we can see that dn4 is taking space from sdd disk which is of 15 gb size. so, let's create partition of 5Gb in this disk so that we can achieve what we want.

To create new partition:

1. we have to first go inside it using the command → fdisk /dev/sdd.
2. Then press n to create a new partition
3. select p for creating primary partition, then give the sectors from which you want to start the partition (it is default selected just press enter)
4. Now give the size of partition you want to create(with + sign) like here i have given 5G.
5. Then write →w to save what we have done

The above process has been shown in the below screenshots👇

You can confirm the newly created partition i.e. sdd2 as shown below👇

Now, the next step is to format this newly created partition for which we can use command → mkfs.ext4 /dev/sdd

Now is the final step which is to mount dn4 folder to this newly created partition which can be done by below command

This is what we have to do. Now you can check your hadoop cluster file system that you now have a new shared storage as below of about 5Gb👇

So, this was what we want to achieve.

This was the task given by vimal sir in the ARTH training which i did with my team members (team 14.9):

1. Rupali Gurjar(Rupali Gurjar)
2. Manmohan
3. kritika sharma
4. and me (Deepanshu Yadav)