Explore Courses
Liverpool Business SchoolLiverpool Business SchoolMBA by Liverpool Business School
  • 18 Months
Bestseller
Golden Gate UniversityGolden Gate UniversityMBA (Master of Business Administration)
  • 15 Months
Popular
O.P.Jindal Global UniversityO.P.Jindal Global UniversityMaster of Business Administration (MBA)
  • 12 Months
New
Birla Institute of Management Technology Birla Institute of Management Technology Post Graduate Diploma in Management (BIMTECH)
  • 24 Months
Liverpool John Moores UniversityLiverpool John Moores UniversityMS in Data Science
  • 18 Months
Popular
IIIT BangaloreIIIT BangalorePost Graduate Programme in Data Science & AI (Executive)
  • 12 Months
Bestseller
Golden Gate UniversityGolden Gate UniversityDBA in Emerging Technologies with concentration in Generative AI
  • 3 Years
upGradupGradData Science Bootcamp with AI
  • 6 Months
New
University of MarylandIIIT BangalorePost Graduate Certificate in Data Science & AI (Executive)
  • 8-8.5 Months
upGradupGradData Science Bootcamp with AI
  • 6 months
Popular
upGrad KnowledgeHutupGrad KnowledgeHutData Engineer Bootcamp
  • Self-Paced
upGradupGradCertificate Course in Business Analytics & Consulting in association with PwC India
  • 06 Months
OP Jindal Global UniversityOP Jindal Global UniversityMaster of Design in User Experience Design
  • 12 Months
Popular
WoolfWoolfMaster of Science in Computer Science
  • 18 Months
New
Jindal Global UniversityJindal Global UniversityMaster of Design in User Experience
  • 12 Months
New
Rushford, GenevaRushford Business SchoolDBA Doctorate in Technology (Computer Science)
  • 36 Months
IIIT BangaloreIIIT BangaloreCloud Computing and DevOps Program (Executive)
  • 8 Months
New
upGrad KnowledgeHutupGrad KnowledgeHutAWS Solutions Architect Certification
  • 32 Hours
upGradupGradFull Stack Software Development Bootcamp
  • 6 Months
Popular
upGradupGradUI/UX Bootcamp
  • 3 Months
upGradupGradCloud Computing Bootcamp
  • 7.5 Months
Golden Gate University Golden Gate University Doctor of Business Administration in Digital Leadership
  • 36 Months
New
Jindal Global UniversityJindal Global UniversityMaster of Design in User Experience
  • 12 Months
New
Golden Gate University Golden Gate University Doctor of Business Administration (DBA)
  • 36 Months
Bestseller
Ecole Supérieure de Gestion et Commerce International ParisEcole Supérieure de Gestion et Commerce International ParisDoctorate of Business Administration (DBA)
  • 36 Months
Rushford, GenevaRushford Business SchoolDoctorate of Business Administration (DBA)
  • 36 Months
KnowledgeHut upGradKnowledgeHut upGradSAFe® 6.0 Certified ScrumMaster (SSM) Training
  • Self-Paced
KnowledgeHut upGradKnowledgeHut upGradPMP® certification
  • Self-Paced
IIM KozhikodeIIM KozhikodeProfessional Certification in HR Management and Analytics
  • 6 Months
Bestseller
Duke CEDuke CEPost Graduate Certificate in Product Management
  • 4-8 Months
Bestseller
upGrad KnowledgeHutupGrad KnowledgeHutLeading SAFe® 6.0 Certification
  • 16 Hours
Popular
upGrad KnowledgeHutupGrad KnowledgeHutCertified ScrumMaster®(CSM) Training
  • 16 Hours
Bestseller
PwCupGrad CampusCertification Program in Financial Modelling & Analysis in association with PwC India
  • 4 Months
upGrad KnowledgeHutupGrad KnowledgeHutSAFe® 6.0 POPM Certification
  • 16 Hours
O.P.Jindal Global UniversityO.P.Jindal Global UniversityMaster of Science in Artificial Intelligence and Data Science
  • 12 Months
Bestseller
Liverpool John Moores University Liverpool John Moores University MS in Machine Learning & AI
  • 18 Months
Popular
Golden Gate UniversityGolden Gate UniversityDBA in Emerging Technologies with concentration in Generative AI
  • 3 Years
IIIT BangaloreIIIT BangaloreExecutive Post Graduate Programme in Machine Learning & AI
  • 13 Months
Bestseller
IIITBIIITBExecutive Program in Generative AI for Leaders
  • 4 Months
upGradupGradAdvanced Certificate Program in GenerativeAI
  • 4 Months
New
IIIT BangaloreIIIT BangalorePost Graduate Certificate in Machine Learning & Deep Learning (Executive)
  • 8 Months
Bestseller
Jindal Global UniversityJindal Global UniversityMaster of Design in User Experience
  • 12 Months
New
Liverpool Business SchoolLiverpool Business SchoolMBA with Marketing Concentration
  • 18 Months
Bestseller
Golden Gate UniversityGolden Gate UniversityMBA with Marketing Concentration
  • 15 Months
Popular
MICAMICAAdvanced Certificate in Digital Marketing and Communication
  • 6 Months
Bestseller
MICAMICAAdvanced Certificate in Brand Communication Management
  • 5 Months
Popular
upGradupGradDigital Marketing Accelerator Program
  • 05 Months
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in Corporate & Financial Law
  • 12 Months
Bestseller
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in AI and Emerging Technologies (Blended Learning Program)
  • 12 Months
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in Intellectual Property & Technology Law
  • 12 Months
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in Dispute Resolution
  • 12 Months
upGradupGradContract Law Certificate Program
  • Self paced
New
ESGCI, ParisESGCI, ParisDoctorate of Business Administration (DBA) from ESGCI, Paris
  • 36 Months
Golden Gate University Golden Gate University Doctor of Business Administration From Golden Gate University, San Francisco
  • 36 Months
Rushford Business SchoolRushford Business SchoolDoctor of Business Administration from Rushford Business School, Switzerland)
  • 36 Months
Edgewood CollegeEdgewood CollegeDoctorate of Business Administration from Edgewood College
  • 24 Months
Golden Gate UniversityGolden Gate UniversityDBA in Emerging Technologies with Concentration in Generative AI
  • 36 Months
Golden Gate University Golden Gate University DBA in Digital Leadership from Golden Gate University, San Francisco
  • 36 Months
Liverpool Business SchoolLiverpool Business SchoolMBA by Liverpool Business School
  • 18 Months
Bestseller
Golden Gate UniversityGolden Gate UniversityMBA (Master of Business Administration)
  • 15 Months
Popular
O.P.Jindal Global UniversityO.P.Jindal Global UniversityMaster of Business Administration (MBA)
  • 12 Months
New
Deakin Business School and Institute of Management Technology, GhaziabadDeakin Business School and IMT, GhaziabadMBA (Master of Business Administration)
  • 12 Months
Liverpool John Moores UniversityLiverpool John Moores UniversityMS in Data Science
  • 18 Months
Bestseller
O.P.Jindal Global UniversityO.P.Jindal Global UniversityMaster of Science in Artificial Intelligence and Data Science
  • 12 Months
Bestseller
IIIT BangaloreIIIT BangalorePost Graduate Programme in Data Science (Executive)
  • 12 Months
Bestseller
O.P.Jindal Global UniversityO.P.Jindal Global UniversityO.P.Jindal Global University
  • 12 Months
WoolfWoolfMaster of Science in Computer Science
  • 18 Months
New
Liverpool John Moores University Liverpool John Moores University MS in Machine Learning & AI
  • 18 Months
Popular
Golden Gate UniversityGolden Gate UniversityDBA in Emerging Technologies with concentration in Generative AI
  • 3 Years
Rushford, GenevaRushford Business SchoolDoctorate of Business Administration (AI/ML)
  • 36 Months
Ecole Supérieure de Gestion et Commerce International ParisEcole Supérieure de Gestion et Commerce International ParisDBA Specialisation in AI & ML
  • 36 Months
Golden Gate University Golden Gate University Doctor of Business Administration (DBA)
  • 36 Months
Bestseller
Ecole Supérieure de Gestion et Commerce International ParisEcole Supérieure de Gestion et Commerce International ParisDoctorate of Business Administration (DBA)
  • 36 Months
Rushford, GenevaRushford Business SchoolDoctorate of Business Administration (DBA)
  • 36 Months
Liverpool Business SchoolLiverpool Business SchoolMBA with Marketing Concentration
  • 18 Months
Bestseller
Golden Gate UniversityGolden Gate UniversityMBA with Marketing Concentration
  • 15 Months
Popular
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in Corporate & Financial Law
  • 12 Months
Bestseller
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in Intellectual Property & Technology Law
  • 12 Months
Jindal Global Law SchoolJindal Global Law SchoolLL.M. in Dispute Resolution
  • 12 Months
IIITBIIITBExecutive Program in Generative AI for Leaders
  • 4 Months
New
IIIT BangaloreIIIT BangaloreExecutive Post Graduate Programme in Machine Learning & AI
  • 13 Months
Bestseller
upGradupGradData Science Bootcamp with AI
  • 6 Months
New
upGradupGradAdvanced Certificate Program in GenerativeAI
  • 4 Months
New
KnowledgeHut upGradKnowledgeHut upGradSAFe® 6.0 Certified ScrumMaster (SSM) Training
  • Self-Paced
upGrad KnowledgeHutupGrad KnowledgeHutCertified ScrumMaster®(CSM) Training
  • 16 Hours
upGrad KnowledgeHutupGrad KnowledgeHutLeading SAFe® 6.0 Certification
  • 16 Hours
KnowledgeHut upGradKnowledgeHut upGradPMP® certification
  • Self-Paced
upGrad KnowledgeHutupGrad KnowledgeHutAWS Solutions Architect Certification
  • 32 Hours
upGrad KnowledgeHutupGrad KnowledgeHutAzure Administrator Certification (AZ-104)
  • 24 Hours
KnowledgeHut upGradKnowledgeHut upGradAWS Cloud Practioner Essentials Certification
  • 1 Week
KnowledgeHut upGradKnowledgeHut upGradAzure Data Engineering Training (DP-203)
  • 1 Week
MICAMICAAdvanced Certificate in Digital Marketing and Communication
  • 6 Months
Bestseller
MICAMICAAdvanced Certificate in Brand Communication Management
  • 5 Months
Popular
IIM KozhikodeIIM KozhikodeProfessional Certification in HR Management and Analytics
  • 6 Months
Bestseller
Duke CEDuke CEPost Graduate Certificate in Product Management
  • 4-8 Months
Bestseller
Loyola Institute of Business Administration (LIBA)Loyola Institute of Business Administration (LIBA)Executive PG Programme in Human Resource Management
  • 11 Months
Popular
Goa Institute of ManagementGoa Institute of ManagementExecutive PG Program in Healthcare Management
  • 11 Months
IMT GhaziabadIMT GhaziabadAdvanced General Management Program
  • 11 Months
Golden Gate UniversityGolden Gate UniversityProfessional Certificate in Global Business Management
  • 6-8 Months
upGradupGradContract Law Certificate Program
  • Self paced
New
IU, GermanyIU, GermanyMaster of Business Administration (90 ECTS)
  • 18 Months
Bestseller
IU, GermanyIU, GermanyMaster in International Management (120 ECTS)
  • 24 Months
Popular
IU, GermanyIU, GermanyB.Sc. Computer Science (180 ECTS)
  • 36 Months
Clark UniversityClark UniversityMaster of Business Administration
  • 23 Months
New
Golden Gate UniversityGolden Gate UniversityMaster of Business Administration
  • 20 Months
Clark University, USClark University, USMS in Project Management
  • 20 Months
New
Edgewood CollegeEdgewood CollegeMaster of Business Administration
  • 23 Months
The American Business SchoolThe American Business SchoolMBA with specialization
  • 23 Months
New
Aivancity ParisAivancity ParisMSc Artificial Intelligence Engineering
  • 24 Months
Aivancity ParisAivancity ParisMSc Data Engineering
  • 24 Months
The American Business SchoolThe American Business SchoolMBA with specialization
  • 23 Months
New
Aivancity ParisAivancity ParisMSc Artificial Intelligence Engineering
  • 24 Months
Aivancity ParisAivancity ParisMSc Data Engineering
  • 24 Months
upGradupGradData Science Bootcamp with AI
  • 6 Months
Popular
upGrad KnowledgeHutupGrad KnowledgeHutData Engineer Bootcamp
  • Self-Paced
upGradupGradFull Stack Software Development Bootcamp
  • 6 Months
Bestseller
KnowledgeHut upGradKnowledgeHut upGradBackend Development Bootcamp
  • Self-Paced
upGradupGradUI/UX Bootcamp
  • 3 Months
upGradupGradCloud Computing Bootcamp
  • 7.5 Months
PwCupGrad CampusCertification Program in Financial Modelling & Analysis in association with PwC India
  • 5 Months
upGrad KnowledgeHutupGrad KnowledgeHutSAFe® 6.0 POPM Certification
  • 16 Hours
upGradupGradDigital Marketing Accelerator Program
  • 05 Months
upGradupGradAdvanced Certificate Program in GenerativeAI
  • 4 Months
New
upGradupGradData Science Bootcamp with AI
  • 6 Months
Popular
upGradupGradFull Stack Software Development Bootcamp
  • 6 Months
Bestseller
upGradupGradUI/UX Bootcamp
  • 3 Months
PwCupGrad CampusCertification Program in Financial Modelling & Analysis in association with PwC India
  • 4 Months
upGradupGradCertificate Course in Business Analytics & Consulting in association with PwC India
  • 06 Months
upGradupGradDigital Marketing Accelerator Program
  • 05 Months

Top 20 HDFS Commands You Should Know About [2024]

Updated on 21 November, 2022

9.32K+ views
7 min read

Hadoop is an Apache open-source structure that enables the distributed processing of large-scale data sets over batches of workstations with simple programming patterns. It operates in a distributed storage environment with numerous clusters of computers with the best scalability features. Read more about HDFS and it’s architecture.

Goals of HDFS

1. It Provides a Large-Scale Distributed File System

10k nodes, 100 million files, and 10 PB

2. Optimization of Batch Processing                    

Provides very comprehensive aggregated capacity

3. Assume Commodity Hardware                                 

It detects hardware failure and recovers it

Possibilities of consuming the existing file if the hardware fails

4. Best Smart Client Intelligence Solution

The client can find the location of the scaffolds

The client can access the data directly from the data nodes      

5. Data Consistency                                                                                             

The client can append to the existing files

It is the Write-once-Read-many access model 

6. Chunks of File Replication and Usability 

Files can be a break in multi-nodes blocks in the 128 MB-block sizes and reuse it

7. Meta-Data in Memory      

The entire Meta-data is stored in the main memory

Meta-data is in the list of files, a list of blocks, and a list of data-nodes

Transaction-logs, it records file creation and file-deletions

8. Data-Correctness

It uses the checksum to validate and transform the data.

Its client calculates the checksum per 512 bytes. The client retrieves the data and its checksum from the nodes

If validations fail, the client can use the replica-process.

9. Data-Pipelining Process    

Its client begins the initial step of writing from the first nodes

The first data-nodes transmit the data to the next data node to the pipeline

When all models are written, the client moves on to the next step to write the next block in the file

HDFS Architecture

Hadoop Distributed File System (HDFS) is structured into blocks. HDFS architecture is described as a master/slave one. Namenode and data node make up the HDFS architecture.

  1. Namenode: It functions as a master server for managing the file system namespace and also provides the right access approach to the clients.
  • It provides all the data nodes comprising data blocks for a particular file. With the help of this, when the system starts, it restores the data from the data nodes every time. 
  • HDFS incorporates a file method namespace that is executed with the Namenode for common operations like file “opening, closing, and renaming”, and even for catalogue.
  1. Datanode: It is the second technique specification in the HDFS cluster. It usually works one per node in the HDFS cluster. 
  • DataNodes are the methods that perform like slaves, stay on each computer in a cluster mode, and implement the original storage. They serve, read, and write requests for the clients.

HDFS Top 20 Commands

Here is a list of all the HDFS commands:

1. To get the list of all the files in the HDFS root directory

  • Command: Usage: hdfs dfs [generic options] -ls [-c] [-h] [-q] [-R] [-t] [-S] [-u] [<path>…]   
  • Note: Here, choose the path from the root, just like the general Linux file system. -h in Green Mark shows that it is in human-readable sizes, as recommended. -R in Blue Mark shows that it is different from numerous one to practice into subdirectories.

2. Help

  • Command: fs –help        
  • Note: It prints the long output which prints all the commands

3. Concatenate all the files into a catalogue within a single file

  • Command: hdfs dfs [generic options] -getmerge [-nl] <src> <localdst>
  • Note: This will generate a new file on the local system directory which carries all files from a root directory and concatenates all together. -nl option, which is marked in Red, combines newlines among the files. With the help of this command, you can combine a collection of small records within a selection for a different operation.

 4. Show Disk Usage in Megabytes for the Register Directory: /dir

  • Command: hdfs dfs [generic options] -du [-s] [-h] <path> …
  • Note: The -h, which is marked in Blue gives you a readable output of size, i.e., Gigabytes.

5. Modifying the replication factor for a file

  • Command: hadoop fs -setrep -w 1 /root/journaldev_bigdata/derby.log
  • Note: It is for replication factors, which count by a file, which can be replicated in each Hadoop cluster. 

6. copyFromLocal

  • Command: hadoop fs -copyFromLocal derby.log /root/journaldev_bigdata
  • Note: This command is for copy of a file from Local file System to Hadoop FS

7.-rm -r

  • Command: hadoop fs -rm -r /root/journaldev_bigdata
  • Note:  With the help of rm-r command, we can remove an entire HDFS directory

8. Expunge

  • Command: hadoop fs -expunge
  • Note: This expunge performs fragments empty.

9. fs -du

  • Command:  hadoop fs -du /root/journaldev_bigdata/
  • Note: This command helps to disk usage of files under HDFS in a directory.

10.mkdir 

  • Command:hadoop fs -mkdir /root/journaldev_bigdata
  • Note: This command is used for checking the health of the files.

11.text

  • Command: hadoop fs -text <src>
  • Note: This command is used to visualise the .“sample zip” file in text format.

12. Stat 

  • Command: hadoop fs -stat [format] <path>
  • Note:  This stat command is used to print the information about the ‘test’ file present in the directory.

13. chmod : (Hadoop chmod Command Usage)

  • Command: hadoop fs -chmod [-R] <mode> <path>
  • Note: This command is used for changing the file permission on “testfile”.

14. appendToFile

  • Command: hadoop fs -appendToFile <localsrc> <dest>
  • Note: This command can be used for appending the localfile1, localfile2 instantly in the local filesystem into the file specified as ‘appendfile’ in the catalog.

15. Checksum

  • Command: hadoop fs -checksum <src>
  • Note: This is the shell command which returns the checksum information.

16.Count

  • Command: hadoop fs -count [options] <path>
  • Note: This command is used for counting the numbers of files, directories, and bytes from the specified path of the given file.

17. Find

  • Command: hadoop fs -find <path> … <expression>
  • Note: This command is used for finding all files which match the mentioned expression.

18. getmerge

  • Command: hadoop fs -getmerge <src> <localdest>
  • Note: This command is used for “MergeFile into Local”.

19. touchz

  • Command: hadoop fs –touchz /directory/filename
  • Note: This command generates a file in HDFS with a file size corresponding to 0 bytes. 

20. fs -ls

  • Command: hadoop fs -ls
  • Note: This command generates a list of  available files and subdirectories under default directory.

Hopefully, this article helped you with understanding HDFS commands to execute operations on the Hadoop filesystem. The article has described all the fundamental HDFS commands.

If you are interested to know more about Big Data, check out our PG Diploma in Software Development Specialization in Big Data program which is designed for working professionals and provides 7+ case studies & projects, covers 14 programming languages & tools, practical hands-on workshops, more than 400 hours of rigorous learning & job placement assistance with top firms.

Learn Software Development Courses online from the World’s top Universities. Earn Executive PG Programs, Advanced Certificate Programs or Masters Programs to fast-track your career.

Frequently Asked Questions (FAQs)

1. What is HDFS and how does it work?

The Hadoop Distributed File System (HDFS) is the storage system for Hadoop spread out over multiple machines as a means to reduce cost and increase reliability. HDFS exposes a file system namespace and user data to be stored in files. Hadoop works on multiple machines simultaneously and huge data is processed across a cluster of commodity servers. With Hadoop, work can be done on multiple machines simultaneously. It stores the data while MapReduce processes the data and YARN divides the tasks. Namenode and datanode make up the Hadoop architecture. In short, we can say that first, the client submits data, HDFS stores the data, and MapReduce processes the data.

2. How are files stored in HDFS?

In HDFS, data is stored in blocks, the smallest unit of data that the file system stores. Files are divided into blocks, and each block is stored on a DataNode. Multiple DataNodes are linked to the master node in the cluster which is called the NameNode. For all files of HDFS, the storage type is defined in the data store. HDFS is also efficient in storing very large files across machines in a large cluster. Each file is stored as a sequence of blocks.

3. What is Apache Pig and its applications?

Pig is a high-level platform which is used to process large datasets. To process the data which is stored in the HDFS, the programmers write the scripts using the Pig Latin Language. With the help of a multi-query approach in Apache Pig, we can reduce the time of development. Pig is also beneficial for programmers who are not from a Java background. It is easy to learn, read, and write, and can handle the analysis of both structured and unstructured data. Pig is used for exploring large datasets and collecting large amounts of datasets in the form of search logs. It is used where the analytical insights are needed using the sampling.