Programming VIP

What needs to be done: a full scale to calculate the average start and end time of the task in seven days 1, Introduction data The data table is a full synchronization table. The partition is based on the date. It contains the start time, end time, total seconds of start time (total seconds to the early morning), and total seconds of end ...

hive distribute by group application tuning Group by fields in the table set hive.auto.convert.join=true; set hive.auto.convert.join.noconditionaltask=true; set hive.auto.convert.join.noconditionaltask.size=10000000; set hive.mapjoin.smalltable.filesize=200000000; set hive.merge.mapfiles = true; set hive.merge.mapredfiles = false; --MR Small ...

Perfect version of kettle to realize data incremental synchronization preface Some time ago, there was an operation of using kettle to realize data synchronization, including Installation and configuration of kettle, creation of job, creation of translate, etc. At that time, the time point of dead writing was used (that is, the data wil ...

Doris storage file format optimization #File format Documents include: The beginning of the file is an 8-byte magic code, which is used to identify the file format and version Data Region: used to store the data information of each column. The data here is loaded by page on demand Index Region: doris uniformly stores the index data of each ...

Basic steps of Flink programming: 1. Create the stream execution environment streamexecutionenvironment Getexecutionenvironment() gets the stream environment. 2. Load data Source 3. Transformation 4. Output Sink, land it in other data warehouses and print it directly Basic operation of Flink data -- four categories Operation of a single ...

Big data warehouse training - task 3 Data analysis and prediction of Taobao double 11 Case introduction The case of Taobao double 11 data analysis and prediction course involves various typical operations involved in the whole process of data processing such as data preprocessing, storage, query and visual analysis, including the installatio ...

Chapter 2 Hive installation 2.1 hive installation address 1. Hive official website address http://hive.apache.org/ 2. Document viewing address https://cwiki.apache.org/confluence/display/Hive/GettingStarted 3. Download address http://archive.apache.org/dist/hive/ 4. github address https://github.com/apache/hive 2.2 Hive installation a ...

Basic operation of function View the description information of the specified function: desc function function name;Display function extension content: desc function extended function name; Typical advanced functions Group sorting takes TopN To implement the grouping sorting function, you need to use row_number and over functions. row_ ...

Hive has built-in HiveServer and HiveServer2 services, both of which allow clients to connect using multiple programming languages. However, HiveServer cannot handle concurrent requests from multiple clients, so HiveServer2 is generated. HiveServer2 (HS2) allows remote clients to submit requests to hive and retrieve results in various programmi ...

Hive tuning: Choosing the appropriate "storage format" and "compression method" for the analyzed data can improve the analysis efficiency of hive Data compression format: When selecting a compression algorithm, you need to consider whether it can be divided, If segmentation is not supported (the integrity of a pi ...