Programming VIP

1, Performance optimization analysis The execution of a computing task mainly depends on CPU, memory and bandwidth. Spark is a memory based computing engine, so for it, the biggest impact may be memory. Generally, our tasks encounter performance bottlenecks, and most of them are memory problems. Of course, CPU and bandwidth may also affect th ...

Since learning Spark requires Scala, here are some basic grammars of scala. be careful: Scala doesn't need a semicolon at the end of a line 1 variable type val is immutable. It must be initialized at the time of declaration, and it cannot be assigned again after initializationvar is variable. It needs to be initialized when declaring. After ...

background In previous articles Analysis and solution of DataSourceScanExec NullPointerException caused by spark DPP , we directly skipped the step of dynamic code generation failure. This time, let's analyze that SQL is still in the article mentioned above. analysis After running the sql, we can see the following physical plan: We can see ...

Git address: https://gitee.com/jyq_18792721831/sparkmaven.git Create Project Let's start by creating a generic maven project Create a project followed by a hello module Is also a normal maven module Increase scala dependency We don't write code in the parent project, the parent project is just for managing the child project, so th ...

Chapter 1 Introduction to Scala 1.1 general Scala is a static type programming language that takes Java virtual machine (JVM) as the running environment and combines the best features of object-oriented and functional programming (static languages need to be compiled in advance, such as Java, c, c + +, dynamic languages such as js). 1) Scala ...

summary Akka is a library developed using Scala. It is based on event driven toolkit for building high concurrency projects Akka characteristics: Provide an asynchronous, non blocking, high-performance event driven programming modelThe built-in fault tolerance mechanism allows the Actor to recover or reset in case of errorLightweight ...

Chisel combinational logic This section describes how to use Chisel components to implement combinatorial logic. This section will demonstrate how three basic Chisel types (UInt, unsigned integer; SInt, signed integer; Bool, Boolean) can be connected and operated. It should be noted that all Chisel variables are declared as Scala val. never ...

Definition of partial function 1) It is a good choice to use disk function when performing logical operations on conditions that meet certain conditions rather than all conditions 2) Encapsulate a group of case statements wrapped in braces as functions, which we call partial functions. It only calculates the parameters that will act on the exec ...

Spark learning notes [1]-scala environment installation and basic syntax    just as the saying goes, if you want to do a good job, you must first use your tools. Spark's development language is not Java but scala. Although they both run on the JVM, the basic characteristics of the two languages are still somewhat different. Here is a ...

Chapter 1 pattern matching Pattern matching in Scala is similar to the switch syntax in Java int i = 10 switch (i) { case 10 : System.out.println("10"); break; case 20 : System.out.println("20"); break; default : System.out.println("other number"); break; } But scala is more powerful ...