Programming VIP

pyspark learning -- 2. pyspark's running method attempt and various sample code attempts Operation method Pycharmrun spark operation in the system: spark submit Start spark task run Sample code Streaming text processing streaming context Stream text word count Error reporting summary Operati ...

Note: Please note that this is just a MR training project. In practical application, do not use MR calculation friend recommendation and calculation program of directed acyclic graph logic. Because MR will need to write intermediate results to disk, disk IO greatly reduces efficiency. Hadoop is a bit b ...

Spark on K8S (spark on kubernetes operator) environment construction and demo process (2) Common problems in the process of Spark Demo (two) How to persist logs in Spark's executor/driver How to configure Spark history server to take effect What does xxxxx webhook do under spark operator namespace ...

In the spark program, jackson is used to do json serialization and deserialization of scala objects. There are java.lang.NoClassDefFoundError and java.lang.AbstractMethodError errors at runtime. After searching the Internet, it is found that the version conflicts between jackson/guava and spark. 1. In idea, by adjusting the order of dependencie ...

1. Business requirements Calculate a cell phone number (base station, dwell time), (current longitude, current latitude) by holding the cell phone number's dwell time log and base station information at each base station The log information generated by connecting the mobile phone to the base station is similar to the following: ...

Master registration mechanism Application registration The previous article has analyzed the initialization process of the SparkContext, and finally sent the registration information of the RegisterApplication type to the Master Now let's see how the Master responds after receiving these messages First, the Master class inhe ...

Background: sometimes we need to define an external data source and use spark sql to process it. There are two benefits: (1) after defining the external data source, it is very simple to use, and the software architecture is clear. It can be used directly through sql. (2) it is easy to divide modules into layers and build them up layer by lay ...

The X-Pack Spark service provides Redis, Cassandra, MongoDB, HBase and RDS storage services with the ability of complex analysis, streaming processing, warehousing and machine learning through external computing resources, so as to better solve user data processing related scenario problems. RDS & polardb sub table archiving to X-Pack S ...

About big file upload thinking Use js to read the file selected in the form form, calculate the md5 value of the file, upload the md5 value to the server, and check whether the file has been uploaded (similar to the second pass function) If the file has not been uploaded, cut it into 1MB blocks according to its size. If it is smaller than 1MB ...

Explanation of Spark (2) operator Article directory Explanation of Spark (2) operator I. wordcount II. Programming model III. use of RDD data sets and operators 1. Three necessary operators 2. Common operators (Cartesian, cogroup, join) 3. Sorting and aggregation calculation I. wordcount B ...