Programming VIP

Using scala implementation, count the number of each word in the file import scala.io.Source object demo14 { def main(args: Array[String]): Unit = { //read file val source = Source.fromFile("Scala/data/words.txt") //Put the read data into a list collection of List[String] type, and each line is used as an element of the li ...

Big data technology AI Flink/Spark/Hadoop / data warehouse, data analysis, interview, source code interpretation and other dry goods learning materials 101 original content official account Pattern matching in Scala is similar to the switch syntax in Java int i = 10 switch (i) { case 10 : System.out.println("10"); break; case 20 ...

Basic concepts In Scala language, trait is used to replace the concept of interface, that is, multiple classes have the same When the trait (feature) of the is selected, the trait (feature) can be independent and declared with the keyword trait. The trait in Scala can have either abstract attributes and methods or concrete attributes and m ...

catalogue 1, SparkSession, DataFrame and Dataset 2, Spark Sql parsing 1. Overall overview 2. sql syntax parsing key objects 3, Spark LogicalPlan 1. Overall overview 2. LogicalPlan class structure system​ 3. Generated by analyzed logicalplan 1, SparkSession, DataFrame and Dataset 1. To use the sparksql function, you need to create a ...

Scala-19: complex WordCount case 1, Case analysis In a simple case, all the data passed in are strings, as shown below: List( "hello", "hello world", "hello scala", "hello spark from scala", "hello flink from scala" ) If the incoming data is not all string, but after a statistical data, such as List( ...

preface I want to try how to run the sql of flink. Last year, I saw that the big brother JarkWu shared it, but I haven't actually operated it. Recently, I'm interested. Original Jarkwu sharing Demo: building streaming applications based on Flink SQL. JarkWu github flink-sql-demo The boss is based on flink1 Version 10 is built, but the rel ...

1 insufficient spark streaming In 2016, Apache Spark launched the Structured Streaming project, a new stream computing engine based on Spark SQL, which allows users to write high-performance stream processing programs as easily as writing batch programs. Structured Streaming is not a simple improvement to Spark Streaming, but a new stre ...

1.partitionBy 1) Function signature def partitionBy(partitioner: Partitioner): RDD[(K, V)] 2) Function description Repartition the data according to the specified Partitioner. Spark's default Partitioner is HashPartitioner Note: partitionBy can only be called when rdd is converted to key value tuple type import org.apache.spark.{HashPartit ...

1. Assembly 1.1 general Anyone who has learned about programming knows that the sentence "program = algorithm + data structure" was put forward by the famous Swiss computer scientist Nicholas Voss, who was also the winner of the Turing Award in 1984. Algorithm refers to a series of effective and general steps of calculation. Algorit ...

1. Branch control if else Let the program execute selectively. There are three kinds of branch control: single branch, double branch and multi branch 1.1 single branch 1) Basic grammar if (Conditional expression) { Execute code block } Description: when the conditional expression is true, the code of {} will be executed. 2) Case // Ent ...