Programming VIP

Git address: https://gitee.com/jyq_18792721831/sparkmaven.git Create Project Let's start by creating a generic maven project Create a project followed by a hello module Is also a normal maven module Increase scala dependency We don't write code in the parent project, the parent project is just for managing the child project, so th ...

SogouQ log analysis Data research and business analysis Sogou lab is used to provide [user query log (SogouQ)] data, and Spark framework is used to encapsulate the data into RDD for business data processing and analysis. 1) . data introduction: The search engine query log database is designed as a collection of Web query log data in ...

background Translate pyspark code that implements a business logic into sparksql to supplement the historical data for the past six months (run by day) based on sparksql; Core Point 1) Translate pyspark to sparksql; 2) Based on sparksql, supplement the historical data of the past half year (run by day); Realization 1) First, pyspark is tra ...

  Article catalogue preface 1, Download and configuration of cluster environment 1. Download hadoop 2. Configure hadoop environment variables Configure hadoop core environment Configure core site xml Configure HDFS site xml Configure mapred site xml Configure yarn site xml Configure workers Disable firewall 2, Clone ...

Experimental resources 1998.csv airports.csv Experimental environment VMware Workstation Ubuntu 16.04 spark-2.4.5 scala-2.12.10 Experimental content "We are sorry to inform you that your flight XXXX from XX to XX has been delayed." I believe many passengers waiting at the airport do not want to hear this sentence. With the gradua ...

Spark learning notes [1]-scala environment installation and basic syntax    just as the saying goes, if you want to do a good job, you must first use your tools. Spark's development language is not Java but scala. Although they both run on the JVM, the basic characteristics of the two languages are still somewhat different. Here is a ...

This article aims to gradually get familiar with the implementation of the overall architecture of hudi through a certain function, and will not discuss the implementation details of the algorithmhudi newcomer, if you have any questions, please correct themspark : version, 3.1.2 hudi : branch, masterTime: 2022/02/06 First EditionObjective: to r ...

Today is the third day of the lunar new year. Monkey Sai Lei Small talk These days, I send her a red envelope every night, a new year's red envelope, and an expression package can be added. I feel that the Chinese New Year is good and there is no new year flavor. My throat hurts when I eat melon seeds. There are many operators in Spark, in ...

Pyspark establishes Spark RDD Each RDD can be divided into multiple partitions. Each partition can be regarded as a data set fragment and can be saved to different nodes in the Spark clusterRDD itself has fault-tolerant mechanism and is a read-only data structure, which can only generate new RDD through transformation; An RDD can be proces ...

1 Spark SQL overview 1.1 what is Spark SQL Spark SQL is a module used by spark to process structured data. It provides two programming abstractions: DataFrame and DataSet, and acts as a distributed SQL query engine. We have learned about Hive, which converts Hive SQL into MapReduce and then submits it to the cluster for execution, which great ...