Hadoop HDFS operation command

View all commands supported by Hadoop HDFS

hadoop fs

List directory and file information

hadoop fs -ls

Circular listing of directory, subdirectory and file information

hadoop fs -lsr

Copy the test.txt of the local file system to the / user / sunlight directory of the HDFS file system

hadoop fs -put test.txt /user/sunlightcs

Copy test.txt from HDFS to the local file system, as opposed to the - put command

hadoop fs -get /user/sunlightcs/test.txt

View the contents of test.txt in HDFS file system

hadoop fs -cat /user/sunlightcs/test.txt

View last 1KB

hadoop fs -tail /user/sunlightcs/test.txt

Delete the test.txt file from the HDFS file system, and the rm command can also delete the empty directory

hadoop fs -rm /user/sunlightcs/test.txt

Delete the / user / sunlight directory and all subdirectories

hadoop fs -rmr /user/sunlightcs

Copying files from the local file system to the HDFS file system is equivalent to the put command

hadoop fs -copyFromLocal test.txt /user/sunlightcs/test.txt

Copying files from the HDFS file system to the local file system is equivalent to the get command

hadoop fs -copyToLocal /user/sunlightcs/test.txt test.txt

Modify the group of the / user / sunlight directory in the HDFS system, and execute the option - R recursively, just like the linux command

hadoop fs -chgrp [-R] /user/sunlightcs

Modify the owner of the / user/sunlightcs directory in the HDFS system, option - R recursively

hadoop fs -chown [-R] /user/sunlightcs

Modify the permissions of / user / sunlight CS directory in HDFS system. MODE can be 3 digits of the corresponding permissions or + / {rwx}. Option - R is executed recursively

hadoop fs -chmod [-R] MODE /user/sunlightcs

View the number of subdirectories, files, file size, file name / directory name in the PATH directory

hadoop fs -count [-q] PATH

Copy files from SRC to DST, if more than one SRC is specified, DST must be a directory

hadoop fs -cp SRC [SRC ...] DST

Displays the size of each file or directory in the directory

hadoop fs -du PATH

Similar to du, when PATH is a directory, the total size of the directory is displayed

hadoop fs -dus PATH

Empty the recycle bin. When a file is deleted, it will first be moved to the temporary directory. Trash /. When the delay time is exceeded, the file will be permanently deleted

hadoop fs -expunge

Gets all the files specified by SRC, merges them into a single file, and writes them to LOCALDST in the local file system. The option addnl adds a line break at the end of each file

hadoop fs -getmerge SRC [SRC ...] LOCALDST [addnl]

View help documents for a [ls] Command

hadoop fs -help ls

Create directory

hadoop fs -mkdir /user/hadoop/examples (directory / directory name)

View block information

hadoop fsck / -files -blocks

View the occupied capacity of the directory

sudo -u hdfs hadoop fs -du -h  /user/hive/

Modify file directory owner and its group - R: recursively modify all file owners in the directory together

sudo -u hdfs hadoop fs -chown -R hive:hive /user/hive/warehouse/tmp_tanjin.db/tmp_startup_c

Find HDFS Hive snapshot capacity

hdfs dfs -du -h /user/hive/warehouse/.snapshot/   

Create Snapshot

sudo -uhdfs hdfs dfs -createSnapshot $dir $snapshotname 

$dir: Snapshot creation directory, $snapshotname: Create snapshot name

Delete snapshot

sudo -uhdfs hdfs dfs -deleteSnapshot $dir $snapshotname

$dir  Directory path not required .snapshot  This level