Programming VIP

1. Debug environment preparation 1.1 Debug code: MR classic introduction case WordCount 1.1.1 Mapper class public class WordCountMapper extends Mapper<LongWritable, Text,Text,LongWritable> { @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String ...

Today is the third day of the lunar new year. Monkey Sai Lei Small talk These days, I send her a red envelope every night, a new year's red envelope, and an expression package can be added. I feel that the Chinese New Year is good and there is no new year flavor. My throat hurts when I eat melon seeds. There are many operators in Spark, in ...

Atlas2.1.0 detailed installation record of big data components based on Apache open source version (test environment) Note: Atlas installation refers to a large number of online materials. This record is only used for future convenience. If there is infringement in this article, please contact immediately. Component version Component nameCom ...

1 Hadoop overview 1.1 what is Hadoop (1) Hadoop is a distributed system infrastructure developed by the Apache foundation (2) It mainly solves the problems of massive data storage and massive data analysis and calculation (3) In a broad sense, Hadoop usually refers to a broader concept - Hadoop ecosystem 1.2 Hadoop advantages (1) High ...

Big data has to mention the most useful weapon Hadoop. This article is the fastest way for you to get started with Hadoop. Hadoop has a quick introduction and a perceptual understanding, which can also be used as a quick index of steps. This article solves the following problems: Understand what Hadoop isWhat is Hadoop used for and how to use ...

1, mapreduce job oom 1. If the task has not been started, an error will be directly oom reported AM journal: 21/05/10 15:15:13 INFO mapreduce.Job: Task Id : attempt_1617064346277_101596_m_000000_1, Status : FAILEDError: Java heap space21/05/10 15:15:16 INFO mapreduce.Job: Task Id : attempt_1617064346277_101596_m_000000_2, Status : FAILEDError: ...

hive partition 1. Primary zoning A partition in Hive is a subdirectory. It is basically consistent with the slice in map. Map slicing is also to improve parallelism. Open the data in the table separately. When you check the data in the table, write the partition information to avoid scanning the whole table; It is an optimized scheme. The pa ...

Some experience on the use of HDFS Write before: I've been working on big data in the company for some time. Take time to sort out the problems encountered and some better optimization methods. 1.HDFS storage multi directory 1.1 production server disk 1.2 on HDFS site Configure multiple directories in the XML file, and pay attention t ...

I don't know if it's good or not. I'll try my best to tell it Data synchronization The previous article talked about using Sqoop to export the data of Mysql and Hdfs to each other. This is the second chapter of offline warehouse. It is about the processing of business data. The basic business data of offline data warehouse are stored in M ...

preface Part of the content is extracted from the training materials of shangsilicon Valley, dark horse and so on 1. Hadoop Archive    HDFS is not good at storing small files, because each file has at least one block, and the metadata of each block will occupy memory in the NameNode. If there are a large number of small files ...