Tutorial: How to Install HDFS on Windows 11

Hadoop Distributed File System (HDFS) is a distributed file system used by big data applications. In this tutorial, we will learn how to install HDFS on Windows 11 machine.

Prerequisites

Before starting the installation process, ensure that the following requirements are met:

  • A Windows 11 machine
  • Java Development Kit (JDK) 8 or higher version installed on the machine
  • Hadoop installation package downloaded from http://hadoop.apache.org/ website

Installation Steps

Follow the below steps to install HDFS on Windows 11 machine:

Step 1: Extract the Hadoop package

Extract the downloaded Hadoop package to a folder on your machine. We will refer to this folder as HADOOP_HOME in the subsequent installation steps.

Step 2: Configure the Environment Variables

To configure the environment variables, follow these steps:

  1. Press the Windows + S key on your keyboard and search for "Environment Variables".
  2. Click on Edit the system environment variables.
  3. In the System Properties window, click on the Environment Variables button.
  4. In the Environment Variables window, under System Variables section, click on the New button.
  5. Enter HADOOP_HOME as the Variable name and the path to extracted Hadoop folder as the Variable value.
  6. Add %HADOOP_HOME%\bin to the Path variable value.

Step 3: Configure Hadoop

To configure Hadoop, follow these steps:

  1. Navigate to the HADOOP_HOME directory using the command prompt.
  2. Navigate to the etc/hadoop folder.
  3. Open the core-site.xml file in a text editor, and add the following code inside the <configuration> tag:
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
  1. Open the hdfs-site.xml file in a text editor, and add the following code inside the <configuration> tag:
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/hadoop/namenode</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/hadoop/datanode</value>
</property>
<property>
    <name>dfs.namenode.http-address</name>
    <value>localhost:50070</value>
</property>

Step 4: Start Hadoop

To start hadoop, follow these steps:

  1. Open a new command prompt window.
  2. Navigate to the HADOOP_HOME directory.
  3. Run the following commands:
bin\hadoop namenode -format
sbin\start-all.cmd
  1. After the successful execution of the above commands, you can access the Hadoop web interface by visiting http://localhost:50070/ in a web browser.

Step 5: Test Hadoop

To test Hadoop, follow these steps:

  1. Open a new command prompt window.
  2. Navigate to the HADOOP_HOME directory.
  3. Run the following command to create a directory in HDFS:
bin\hadoop fs -mkdir /test
  1. Run the following command to upload a file to HDFS:
bin\hadoop fs -put README.txt /test
  1. After the successful execution of the above commands, list the contents of /test directory using the following command:
bin\hadoop fs -ls /test

You should see README.txt in the /test directory.

Congratulations! You have successfully installed HDFS on your Windows 11 machine.