HDFS - Hadoop Distributed File System

Photo by Mr Cup / Fabien Barral on Unsplash

Functions of a File System

Following are the basic functions of a file system

  1. It controls how data is stored and retrieved.
  2. Maintain metadata about data(files and folders).
  3. Permissions and security to view the files and folders.
  4. Manage storage capacity efficiently.

Different File Systems

Image by Taimoor Yousaf

Local File System vs. HDFS

Image by Taimoor Yousaf

Local File System

Now, Local file system runs on node level. HDFS uses local file system to store files/data on individual nodes. But it has only information of blocks in its local nodes only.

HDFS

On the other hand, HDFS operates on cluster level. So, it has the information which nodes have which blocks of file to process.

Example

You want to store a file of say 6 GB on hadoop cluster. HDFS will breakdown the file into 128MB small blocks spread and replicate it on multiple nodes. Now lets get down to node level, each node have multiple blocks and local file system of node keeps information of the blocks on its local level. It does not have any information what resides on the other node. So, HDFS comes into play and keeps track of all the blocks in different nodes and helps in processing.

Benefits of HDFS

  1. Supports distributed processing by saving data into blocks, not as a whole file.
  2. Handle failures by replicating blocks.
  3. Scalability to support future expansions.
  4. Cost-effective because it uses commodity hardware.

Conclusion:

Did you find this article useful? Give it a clap 👏, share with the community, have some thoughts, or I missed something? Share with me in the comments 📝.

Connect

The author is an data scientist with a passion to build meaningful impact-oriented products. He is Kaggle expert. He is former Google Developer Student Club Lead(GDSC) and AWS educate cloud ambassador. He loves to connect with people. If you like his work, Say Hi to him.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Asim Zahid

Asim Zahid

I can brew up algorithms with a pinch of math, an ounce of Python and piles of data to power your business applications.