Programming VIP

Spark independent cluster (just understand), how spark runs on Yan Cluster mode Here is just a record of how Spark Standalone -- independent cluster mode is built The standalone model is generally not applicable in the company, because the company generally has yarn and does not need to develop two resource management frameworks So there is n ...

1, Performance optimization analysis The execution of a computing task mainly depends on CPU, memory and bandwidth. Spark is a memory based computing engine, so for it, the biggest impact may be memory. Generally, our tasks encounter performance bottlenecks, and most of them are memory problems. Of course, CPU and bandwidth may also affect th ...

Author: Han Xinzi@ShowMeAI Tutorial address: http://www.showmeai.tech/tutorials/84 Article address: http://www.showmeai.tech/article-detail/180 Notice: All Rights Reserved. Please contact the platform and the author for reprint and indicate the source 1.Spark machine learning workflow 1) Spark mllib and ml Spark also has MLlib/ML for big d ...

Author: Han Xinzi@ShowMeAITutorial address: http://www.showmeai.tech/tutorials/84Article address: http://www.showmeai.tech/article-detail/178Notice: All Rights Reserved. Please contact the platform and the author for reprint and indicate the sourceintroductionThis is one of the most widely used cases of video and audio data processing of HDFS, ...

Author: Han Xinzi@ShowMeAITutorial address: http://www.showmeai.tech/tutorials/84Article address: http://www.showmeai.tech/article-detail/176Notice: All Rights Reserved. Please contact the platform and the author for reprint and indicate the sourceintroduction2020, since covid-19 has changed the world and affects everyone's life, this case comb ...

Since learning Spark requires Scala, here are some basic grammars of scala. be careful: Scala doesn't need a semicolon at the end of a line 1 variable type val is immutable. It must be initialized at the time of declaration, and it cannot be assigned again after initializationvar is variable. It needs to be initialized when declaring. After ...

background In previous articles Analysis and solution of DataSourceScanExec NullPointerException caused by spark DPP , we directly skipped the step of dynamic code generation failure. This time, let's analyze that SQL is still in the article mentioned above. analysis After running the sql, we can see the following physical plan: We can see ...

Full steps of Douban movie big data project 1. Douban reptile: When I started to write Douban TV series crawler, I thought it was very simple, but in practice, there was an IP sealing situation, which led to my distress for a long time, and now I finally wrote it No more nonsense, just go to the code: The run function is ...

catalogue Initialize Spark streaming program 1, SparkSql parameter tuning settings 1. Set session time zone 2. Sets the maximum number of bytes a single partition can hold when reading a file 3. Set the threshold for merging small files 4. Sets the number of partitions to use when shuffling data with join or aggregate 5. Set the maximum ...

What is RDDs The full English name is Resilient Distributed Datasets, which translates elastic distributed datasets The Spark The Definitive Guide describes as follows: RDD represents an immutable, partitioned collection of records that can be operated on in parallel. In my personal understanding, rdd is a kind of distributed object collection ...