API operations for Hadoop - HDFS

Little scum · Posted on 7/5/2019 4:33:58 PM

1. Introduce dependence

Note: If you need to manually import the jar package, the HDFS jar package ---- the Hadoop installation directory under share

2. Description of development under window

It is recommended to develop Hadoop applications under Linux, and there will be no compatibility issues. If you want to develop a client application on Windows, you need to set up the following environment:

A. Unzip a Hadoop installation package in a directory in Windows

B. Replace the lib and bin directories under the installation package with the local libraries compiled for the corresponding Windows version of the platform

C. Configure the HADOOP_HOME in the window system to point to the installation package you unzipped

D. Add Hadoop's bin directory to the path variable of Windows system

To operate HDFS in Java, you first need to get a client instance

Login is visible.

Our operation target is HDFS, so the obtained fs object should be an instance of DistributedFileSystem.

Get method: Where does the specific instance client class come from?

- Judging from the configuration value of one of the parameters fs.defaultFS in conf;

If we do not specify fs.defaultFS in our code, and there is no corresponding configuration given under the project classpath, the default value in conf comes from the core-default.xml in the hadoop jar package, and the default value is: file:///, then the acquisition will not be an instance of DistributedFileSystem, but a client object of the local file system;

Addition, deletion, modification and inspection of documents

Login is visible.

Access HDFS via stream

Login is visible.

[Source] API operations for Hadoop - HDFS

Related Posts

Sections viewed