Arrays, mappings, tuples, collections of spark notes

1.1. Array 1.1.1. Fixed-length and variable-length arrays

(1) Definition format of fixed-length array:

val arr=new ArrayT

(2) Variable-length array definition format:

val arr = ArrayBuffer[T]()

Note the need for a guide: import scala. collection. mutable. Array Buffer

package cn.itcast.scala

import scala.collection.mutable.ArrayBuffer

object ArrayDemo {

  def main(args: Array[String]) {

 

    //Initialize a fixed-length array of 8 with all elements of 0

    val arr1 = new Array[Int](8)

    //Print a fixed-length array directly with hashcode values for the array

    println(arr1)

    //Converting an array to an array buffer allows you to see the contents of the original array.

    //toBuffer converts arrays into long array buffers

    println(arr1.toBuffer)

 

    //Note: If new, it's equivalent to calling the array's apply method to assign values directly to the array

    //Initialize a fixed-length array of length 1

    val arr2 = Array[Int](10)

    println(arr2.toBuffer)

 

    //Define a fixed-length array of length 3

    val arr3 = Array("hadoop", "storm", "spark")

    //Use () to access elements

    println(arr3(2))

 

    //Variable length array (array buffer)

    //If you want to use array buffer, you need to import the import scala.collection.mutable.ArrayBuffer package

    val ab = ArrayBuffer[Int]()

    //Append an element to the end of the array buffer

    //+= tail additive element

    ab += 1

    //Adding multiple elements

    ab += (2, 3, 4, 5)

    //Append an array++=

    ab ++= Array(6, 7)

    //Adding an array buffer

    ab ++= ArrayBuffer(8,9)

    //Print array buffer ab

 

    //Insert elements at a certain position in the array with insert, insert elements from a subscript

    ab.insert(0, -1, 0)

    //Delete elements at a location in an array with remove

    ab.remove(0)

    println(ab)

 

  }

}

1.1.2. Traversal arrays

1. Enhance for cycle

2. Useful until generates footmarks, 0 until 10 contains 0 but not 10

package cn.itcast.scala

object ForArrayDemo {

  def main(args: Array[String]) {

    //Initialize an array

    val arr = Array(1,2,3,4,5,6,7,8)

    //Enhanced for cycle

    for(i <- arr)

      println(i)

 

    //A good until generates a Range

    //Reverse is the reverse of the Range generated earlier

    for(i <- (0 until arr.length).reverse)

      println(arr(i))

  }

}

1.1.3. Array Conversion

The field keyword converts the original array to produce a new array, with the original array unchanged

package cn.itcast.scala

 

object ArrayYieldDemo {

  def main(args: Array[String]) {

    //Define an array

    val arr = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)

    //Multiply the even number by 10 and regenerate it into a new array

    val res = for (e <- arr if e % 2 == 0) yield e * 10

    println(res.toBuffer)

 

    //More advanced writing, more refreshing

    //A filter is a filter that receives a function whose return value is boolean.

    //map is equivalent to taking out every element of an array and applying the function passed in.

    val r = arr.filter(_ % 2 == 0).map(_ * 10)

    println(r.toBuffer)

 

  }

}

1.1.4. Common Array Algorithms

In Scala, some methods on arrays are very convenient to operate on arrays.

1.2. Mapping

In Scala, the data structure of hash tables is called mapping.

1.2.1. Building mappings

(1) Constructing mapping format
1. Val map = Map (key - > value, key - > value...)

2. Use tuples to construct Val map = Map ((key, value), (key, value), (key, value)...

1.2.2. Get and modify values in maps

(1) Get the values in the mapping:

Value = map (key)

Useful getOrElse

Note: In Scala, there are two kinds of Maps, one is the Map under the immutable package, the content of which is immutable; the other is the Map under the mutable package, the content of which is variable.

Example:

Note: Usually when we create a collection, we modify a variable with the key val (equivalent to final in java), which means that the reference to the variable is invariable, depending on the type of collection the reference points to.

1.3. Tuples

Mapping is a set of K/V dualities, duality is the simplest form of tuples, tuples can contain many different types of values.

1.3.1. Create tuples

(1) Tuples are aggregates of different types of values; duality is the simplest tuple.

(2) Tuples represent tuples by enclosing different values in parentheses.

Create tuple formats:

val tuple= (element, element...)

1.3.2. Get values in tuples

(1) Get the value format in tuples:

Use underscores and footmarks, such as t. _1 t. _2 t. _3

Note: Element footmarks in tuples start with 1

1.3.3. Converting a dual set into a mapping

Converting a dual set into a mapping:

Call its toMap method

1.3.4. Zipper operation

1. Multiple values can be bound together using the zip command

Note: If the number of elements in the two arrays is not the same, the length of the zipper-operated array is the smaller number of elements in that array.

2. If the number of one element is relatively small, you can use zipAll to fill it with the default element

1.4. Collection

Scala's set has three main types: Seq, Set and Map. All sets extend from Iterable. There are mutable and immutable types in Scala. The set of immutable types can not be changed after initialization (note the difference between the variables modified by val).

1.4.1. List

(1) immutable sequence import scala.collection.immutable._

In Scala, the list is either empty (Nil stands for empty list) or a head element plus a tail list.

9:: List (5, 2): The operator is to create a new list for a given head and tail.

Note: Operators are right-bound, such as 9:: 5:: 2:: Nil is equivalent to 9:: (5:: (2:: Nil)

Common operators of list:

+ (elem: A): List[A] adds an element at the head of the list

(x: A): List[A]. Add an element at the head of the list

+ (elem: A): List[A] adds an element at the end of the list

++ [B] (that: GenTraversable Once [B]): List [B] adds another list from the end of the list

:: (prefix: List[A]): List[A] adds another list at the head of the list

val left = List(1,2,3)

val right = List(4,5,6)

// The following operations are equivalent

left ++ right      // List(1,2,3,4,5,6)

right.:::(left)     // List(1,2,3,4,5,6)

// The following operations are equivalent

0 +: left    //List(0,1,2,3)

left.+:(0)   //List(0,1,2,3)

// The following operations are equivalent

left :+ 4    //List(1,2,3,4)

left.:+(4)   //List(1,2,3,4)

// The following operations are equivalent

0 :: left      //List(0,1,2,3)

left.::(0)     //List(0,1,2,3)

Example:

package cn.itcast.collect

/**

 

 * Invariant List Set Operation

 */

object ImmutListDemo {

  def main(args: Array[String]) {

    //Create an immutable collection

    val lst1 = List(1,2,3)

    //Supplementary: Another way to define list

    val other_lst=2::Nil

    //Get the first element of the collection

    val first=lst1.head

    //Gets a collection of elements in a collection other than the first element.

    val tail=lst1.tail

    //Supplement: If there is only one element in the List, its head is the element, and its tail is Nil.

    println(other_lst.head+"----"+other_lst.tail)

    //Insert 0 in front of lst1 to generate a new List

    val lst2 = 0 :: lst1

    val lst3 = lst1.::(0)

    val lst4 = 0 +: lst1

    val lst5 = lst1.+:(0)

     

 

//Adding an element to lst1 produces a new collection

    val lst6 = lst1 :+ 3

    val lst0 = List(4,5,6)       

    //Merge two lists into a new List

    val lst7 = lst1 ++ lst0

    //Insert lst0 in front of lst1 to generate a new collection

    val lst8 = lst1 ++: lst0

    //Insert lst0 in front of lst1 to generate a new collection

    val lst9 = lst1.:::(lst0)

 

    println(other_lst)

    println(lst1)

    println(first)

    println(tail)

    println(lst2)

    println(lst3)

    println(lst4)

    println(lst5)

    println(lst6)

    println(lst7)

    println(lst8)

    println(lst9)

  }

}

(2) Variable sequence import scala.collection.mutable._

ackage cn.itcast.collect

import scala.collection.mutable.ListBuffer

 

object MutListDemo extends App{

  //Build a variable list with three initial elements 1, 2, 3

  val lst0 = ListBuffer[Int](1,2,3)

  //Create an empty variable list

  val lst1 = new ListBuffer[Int]

  //Adding elements to lst1, note that no new collection is generated

  lst1 += 4

  lst1.append(5)

 

 

  //Recent the elements in lst1 to lst0. Note that no new collection has been generated

  lst0 ++= lst1

 

 

  //Merge lst0 and lst1 into a new ListBuffer Note: Generate a collection

  val lst2= lst0 ++ lst1

 

 

  //Append elements to lst0 to generate a new set

  val lst3 = lst0 :+ 5

 

 

  //Delete elements, note: no new collection is generated

  val lst4 = ListBuffer[Int](1,2,3,4,5)

  lst4 -= 5

 

 

  //Delete a list of collections and generate a new collection

  val lst5=lst4--List(1,2)

 

 

  //To convert a variable list into an immutable list, add toList directly

  val lst6=lst5.toList

 

 

  //Using toArray to transform an array of variable list s

  val lst7=lst5.toArray

 

 

  println(lst0)

  println(lst1)

  println(lst2)

  println(lst3)

  println(lst4)

  println(lst5)

  println(lst6)

  println(lst7)

 

}

1.4.2. Set

(1) Immutable Set import scala.collection.immutable._

Set represents a collection without duplicate elements; it is useless to add duplicate elements to Set, and Set does not guarantee the order of insertion, that is, the elements in Set are disorderly.

Definition: Val set = Set (element, element,....))

//Define an immutable Set set Set

 

scala> val set =Set(1,2,3,4,5,6,7)

 

set: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 7, 3, 4)

 

  

 

//Number of elements

 

scala> set.size

 

res0: Int = 7

 

  

 

//Minimum Set

 

scala> set.min

 

res1: Int = 1

 

  

 

//Take the maximum of the set

 

scala> set.max

 

res2: Int = 7

 

  

 

//Merge elements with set1 to create a new set, with the original set unchanged

 

scala> set + 8

 

res3: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 7, 3, 8, 4)

 

  

 

scala> val set1=Set(7,8,9)

 

set1: scala.collection.immutable.Set[Int] = Set(7, 8, 9)

 

  

 

//The intersection of two sets

 

scala> set & set1

 

res4: scala.collection.immutable.Set[Int] = Set(7)

 

  

 

//Union of two sets

 

scala> set ++ set1

 

res5: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 9, 2, 7, 3, 8, 4)

 

  

 

//Remove elements from the second set based on the first set

 

scala> set -- set1

 

res6: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)

 

  

 

//Returns the first set of elements different from the second set

 

scala> set &~ set1

 

res7: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)

 

  

 

  

 

//Calculate the number of eligible elements

 

scala> set.count(_ >5)

 

res8: Int = 2

 

  

 

/Returns the first set of elements different from the second

 

scala> set.diff(set1)

 

res9: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)

 

  

 

/Returns the first set of elements different from the second

 

scala> set1.diff(set)

 

res10: scala.collection.immutable.Set[Int] = Set(8, 9)

 

  

 

//Take the child set(2,5 is the element position, starting from 0, including the head and not the tail)

 

scala> set.slice(2,5)

 

res11: scala.collection.immutable.Set[Int] = Set(6, 2, 7)

 

  

 

//Iterate all the sub-set s, taking the specified number of combinations

 

scala> set1.subsets(2).foreach(x=>println(x))

 

Set(7, 8)

 

Set(7, 9)

 

(2) Variable Set import scala. collection. mutable._

//Import package

 

scala> import scala.collection.mutable

 

import scala.collection.mutable

 

//Define a mutable Set

 

scala> val set1=new HashSet[Int]()

 

set1: scala.collection.mutable.HashSet[Int] = Set()

 

  

 

//Adding elements

 

scala> set1 += 1

 

res1: set1.type = Set(1)

 

  

 

//Adding element add s is equivalent to+=

 

scala> set1.add(2)

 

res2: Boolean = true

 

scala> set1

 

res3: scala.collection.mutable.HashSet[Int] = Set(1, 2)

 

  

 

//Adding a collection of elements to a collection

 

scala> set1 ++=Set(1,4,5)

 

res5: set1.type = Set(1, 5, 2, 4)

 

  

 

//Delete an element

 

scala> set1 -=5

 

res6: set1.type = Set(1, 2, 4)

 

  

 

//Delete an element

 

scala> set1.remove(1)

 

res7: Boolean = true

 

scala> set1

 

res8: scala.collection.mutable.HashSet[Int] = Set(2, 4)

1.4.3. Map

(1) Immutable Map import scala. collection. immutable._

Definition Map aggregate

 

1.val map=Map(key -> value , key -> value...)

 

2.Utilizing tuple construction  val map=Map((Key, value), (Key, value) , (Key, value)....)

 

//Form of presentation:

 

val  map = Map("zhangsan"->30,"lisi"->40)

 

val  map = Map(("zhangsan",30),("lisi",40))

 

  

 

3.operation map aggregate

 

//Get the value: value = map (key)

 

//Principle: By obtaining the key first, the corresponding value of the key is obtained.

 

  

 

4.ergodic map aggregate

 

scala> val imap=Map("zhangsan" -> 20,"lisi" ->30)

 

imap: scala.collection.immutable.Map[String,Int] = Map(zhangsan -> 20, lisi -> 30)

 

//Method 1: Display all key s

 

scala> imap.keys

 

res0: Iterable[String] = Set(zhangsan, lisi)

 

  

 

//Method 2: Display all key s

 

scala> imap.keySet

 

res1: scala.collection.immutable.Set[String] = Set(zhangsan, lisi)

 

  

 

//Get value by key

 

scala> imap("lisi")

 

res2: Int = 30

 

  

 

//Getting the value corresponding to the key through the key returns, and returns the default value of 0 if not.

 

scala> imap.getOrElse("zhangsan",0)

 

res4: Int = 20

 

  

 

//Without the corresponding key, return the default 0

 

scala> imap.getOrElse("zhangsan1",0)

 

res5: Int = 0

 

//Because it is an immutable map, you cannot add, delete or modify key-value pairs to it.

(2) Variable Map import scala. collection. mutable._

//Guide bag

 

import scala.collection.mutable

 

//Declare a variable set

 

scala> val user =mutable.HashMap("zhangsan"->50,"lisi" -> 100)

 

user: scala.collection.mutable.HashMap[String,Int] = Map(lisi -> 100, zhangsan -> 50)

 

  

 

//Adding key-value pairs

 

scala> user +=("wangwu" -> 30)

 

res0: user.type = Map(lisi -> 100, zhangsan -> 50, wangwu -> 30)

 

  

 

//Adding multiple key-value pairs

 

scala> user += ("zhangsan0" -> 30,"lisi0" -> 20)

 

res1: user.type = Map(zhangsan0 -> 30, lisi -> 100, zhangsan -> 50, lisi0 -> 20,wangwu -> 30)

 

  

 

//Method 1: Display all key s

 

scala> user.keys

 

res2: Iterable[String] = Set(zhangsan0, lisi, zhangsan, lisi0, wangwu)

 

  

 

//Method 2: Display all key s

 

scala> user.keySet

 

res3: scala.collection.Set[String] = Set(zhangsan0, lisi, zhangsan, lisi0, wangwu)

 

  

 

//Get value by key

 

scala> user("zhangsan")

 

res4: Int = 50

 

  

 

//Getting the value corresponding to the key through the key returns, and returns the default value of 0 if not.

 

scala> user.getOrElse("zhangsan",0)

 

res5: Int = 50

 

  

 

//Without the corresponding key, return the default 0

 

scala> user.getOrElse("zhangsan1",0)

 

res6: Int = 0

 

  

 

//Update key-value pairs

 

scala> user("zhangsan") = 55

 

scala> user("zhangsan")

 

res8: Int = 55

 

  

 

//Update multiple key-value pairs

 

scala> user += ("zhangsan" -> 60, "lisi" -> 50)

 

res9: user.type = Map(zhangsan0 -> 30, lisi -> 50, zhangsan -> 60, lisi0 -> 20,wangwu -> 30)

 

  

 

//Delete key

 

scala> user -=("zhangsan")

 

res14: user.type = Map(zhangsan0 -> 30, lisi -> 50, lisi0 -> 20, wangwu -> 30)

 

  

 

//Delete key

 

scala>user.remove("zhangsan0")

 

  

 

//Traversal map method 1: Pass the key value

 

scala> for(x<- user.keys) println(x+" -> "+user(x))

 

lisi -> 50

 

lisi0 -> 20

 

wangwu -> 30

 

  

 

//Traversal map method 2: pattern matching

 

scala> for((x,y) <- user) println(x+" -> "+y)

 

lisi -> 50

 

lisi0 -> 20

 

wangwu -> 30

 

  

 

//Traversal map method 3: through foreach

 

scala>  user.foreach{case (x,y) => println(x+" -> "+y)}

 

lisi -> 50

 

lisi0 -> 20

 

wangwu -> 30

Keywords: Big Data Scala Hadoop Spark Java

Added by Avi on Tue, 13 Aug 2019 10:14:47 +0300