Programming VIP

The mechanism of hadoop is only available in hadoop 2.x. the implementation of this function depends on a distributed component: zookeeper. Brief introduction to zookeeper zookeeper mainly provides distributed coordination services. Main functions: 1. Provide storage and management of a small amount of data. 2. Provide monitoring function for d ...

Article directory 1, Introduction to Oozie framework 2, Main functions of Oozie 3, Oozie internal analysis 4, Horizontal and vertical scalability of Oozie 5, The Action execution model of Oozie 1, Introduction to Oozie framework Definition of Oozie: tamer An open source framework based on workfl ...

Data backup of HBase 1.1 backup the table based on the class provided by HBase Use the class provided by HBase to export the data of a table in HBase to HDFS, and then to the test HBase table. (1) = = export from hbase table to HDFS== [hadoop@node01 shells]$ hbase org.apache.hadoop.hbase.mapreduce.Export myuser /hbase_data/myuser_bak (2 ...

Compile the Flink 1.9.0 report, Flink FS Hadoop shaded cannot be found 1.Flink source code download git clone git@github.com:apache/flink.git Then you can switch to different branches of the project and execute the following command to switch the code to release-1.9 branch: git checkout release-1.9 ...

For version 2.1.6, the use of coprocessor requires the introduction of HBase common dependency. Next, record the detailed steps of debugging the test coprocessor   Case background There are two tables, user and people. Each table has a person column family. Now, before inserting "people:name ...

I can't understand a pure technical article so long ago. I suddenly said that it contains sensitive words. I got off the shelf and asked me to resend it... ------   In the process of writing Hadoop MapReduce program, sometimes we need to use some shared global data in mappers or reducers. For exam ...

Hadoop HDFS operation command View all commands supported by Hadoop HDFS hadoop fs List directory and file information hadoop fs -ls Circular listing of directory, subdirectory and file information hadoop fs -lsr Copy the test.txt of the local file system to the / user / sunlight directory ...

Note: Please note that this is just a MR training project. In practical application, do not use MR calculation friend recommendation and calculation program of directed acyclic graph logic. Because MR will need to write intermediate results to disk, disk IO greatly reduces efficiency. Hadoop is a bit b ...

1, Task scheduling overview 1. Why workflow scheduling system is needed 1) A complete data analysis system is usually composed of a large number of task units:shell script program, java program, mapreduce program, hive script, etc. 2) There are time sequence and pre and post dependence relationships ...

Spark on K8S (spark on kubernetes operator) environment construction and demo process (2) Common problems in the process of Spark Demo (two) How to persist logs in Spark's executor/driver How to configure Spark history server to take effect What does xxxxx webhook do under spark operator namespace ...