hadoop familiar with common HDFS operation IDEA installation

Familiar with common HDFS operation 1. Create a txt file in the "/ home/hadoop /" directory of the local Linux file system, where you can enter some words at will 2. View file location locally (ls) 3. Display file content locally cd /home/hadoop ls touch test.txt vim test.txt cat test.txt 4. Use the command to upload the "t ...

Added by alexjb on Thu, 17 Feb 2022 11:21:35 +0200

Hadoop cluster ecological construction

Hadoop cluster construction (continuous update) The relevant resource files that are not used in this paper, the extraction code eeee 1: Preparations to be completed before starting construction Built Linux serverYou can access the public network (ping www.baidu.com), and you can ping itXshell connection (can be omitted)Server version infor ...

Added by ijug.net on Thu, 17 Feb 2022 10:50:58 +0200

07. Introduction, construction and simple use of Hadoop framework Zookeeper

Introduction, construction and simple use of Hadoop framework Zookeeper Introduction to Zookeeper    Zookeeper is an efficient distributed coordination service, which can provide configuration information management, naming, distributed synchronization, cluster management, database switching and other services. It is not suitabl ...

Added by sameerni on Sat, 12 Feb 2022 04:23:08 +0200

python parallel scheduling spark tasks

background Translate pyspark code that implements a business logic into sparksql to supplement the historical data for the past six months (run by day) based on sparksql; Core Point 1) Translate pyspark to sparksql; 2) Based on sparksql, supplement the historical data of the past half year (run by day); Realization 1) First, pyspark is tra ...

Added by crimsonmoon on Fri, 11 Feb 2022 03:30:23 +0200

hadoop cluster building on mac

1. Environmental Instructions Environment & SoftwareEditionOperating System & MacOs11.2.1Virtual Machine & VMware Fusion12.1Server & CentOS7.8jdk1.8hadoop2.9.2 2. Virtual Machine Preparation 1. Configure static ip 1.1 Virtual Machine Network Settings (NAT) VMWARE Fusion sets the virtual machine network adapter link mode to & ...

Added by beyers on Thu, 10 Feb 2022 02:12:24 +0200

CENTOS installs CDH and Kudu-1

edition: CENTOS7.2CDH5.10Kudu1.2 2.1 general This article centos7 2. The process of operating system deploying CDH Enterprise Edition. The installation of Cloudera enterprise data center is mainly divided into four steps: Cluster server configuration, including installing operating system, closing firewall, synchronizing server clock, etc ...

Added by bdata28 on Thu, 10 Feb 2022 01:07:22 +0200

Installation and configuration of Hbase

1, Task structure 2, Test purpose and requirements 1. Experimental purpose 1. Installation and verification of high availability fully distributed mode of HBase 2. Complete the installation of high availability full distribution mode of HBase 3. The related service processes of HBase can be started normally 4. HBase console can be used ...

Added by digibrain on Wed, 09 Feb 2022 14:29:03 +0200

Big data journey for beginners of strange upgrade < Hadoop compression >

Xiaobai's big data journey (57) Hadoop compression Last review After introducing zookeeper, the next step is Hadoop's extended knowledge point, compression and ha. Because HA is based on zookeeper, I bring this knowledge point now Hadoop compression Compression overview First of all, we should know that compression is an optimization meth ...

Added by ozzthegod on Tue, 08 Feb 2022 18:04:45 +0200

Hadoop + spark big data analysis: Hadoop cluster construction

  Article catalogue preface 1, Download and configuration of cluster environment 1. Download hadoop 2. Configure hadoop environment variables Configure hadoop core environment Configure core site xml Configure HDFS site xml Configure mapred site xml Configure yarn site xml Configure workers Disable firewall 2, Clone ...

Added by jonniejoejonson on Tue, 08 Feb 2022 05:25:06 +0200

Construction of Hadoop running environment

1.1 template virtual machine environment preparation 0) install the template virtual machine, with IP address 192.168.10.100, host name Hadoop 100, memory 4G and hard disk 50G 1) The configuration requirements of Hadoop 100 virtual machine are as follows (the Linux system in this paper takes CentOS-7.5-x86-1804 as an example) (1) Using Yu ...

Added by misslilbit02 on Sun, 06 Feb 2022 09:59:58 +0200