Spark Scala 프로그래밍 팁

데이터프레임과 SparkSQL 연동

https://rklicksolutions.wordpress.com/2016/03/03/tutorial-spark-1-6-sql-and-dataframe-operations/

def main(args: Array[String]) {

 

import sqlContext.implicits._

 

val input = sqlContext.read.json("src/main/resources/cars1.json")

 

input.registerTempTable("Cars1")

 

val result = sqlContext.sql("SELECT * FROM Cars1")

 

result.show()

}

 

}

JSON 파서 (Json4s)

http://json4s.org/

강제 형변환

https://stackoverflow.com/questions/6714967/how-do-i-do-casting-in-scala

scala> val myDouble = 25.7
 myDouble: Double = 25.7

 scala> val myInt = 5
 myInt: Int = 5

 scala> val divide = (myDouble / myInt).asInstanceOf[Int]
 divide: Int = 5

List를 DataFrame으로 변환하는 방법

https://stackoverflow.com/questions/41228602/how-to-convert-list-of-tuple-to-dataframe-in-scala

scala> val myList = List(("a1", "a2", "a3"), ("b1", "b2", "b3"), ("c1", "c2", "c3"))
myList: List[(String, String, String)] = List((a1,a2,a3), (b1,b2,b3), (c1,c2,c3))

scala> myList.toDF("col1", "col2", "col3").show
+----+----+----+
|col1|col2|col3|
+----+----+----+
|  a1|  a2|  a3|
|  b1|  b2|  b3|
|  c1|  c2|  c3|
+----+----+----+

ArrayBuffer 삭제 방법

https://alvinalexander.com/scala/how-to-delete-array-arraybuffer-elements-scala-cookbook

scala> var a = ArrayBuffer(1,2,3,4,5)
a: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 2, 3, 4, 5)

scala> a.clear

scala> a
res0: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer()

Scala 조건문, 반복문

http://devcosin.tistory.com/11

Scala 배열

http://changpd.blogspot.kr/2015/05/scala-3.html

1. 고정길이 : Array

scala> val nums = new Array[Int](10)
nums: Array[Int] = Array(0, 0, 0, 0, 0, 0, 0, 0, 0, 0)

scala> val s = Array("hello","world");
s: Array[String] = Array(hello, world)

2. 가변길이 : ArrayBuffer

scala> val b = ArrayBuffer[Int]()
b: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer()

Scala 문자열 연산

https://www.tutorialspoint.com/scala/scala_strings.htm

Scala 삼항연산자

https://programmers.co.kr/learn/courses/30%EB%B6%84-scala/lessons/if%EB%AC%B8-63693ccd-356e-4038-888d-00acf06276c2

val likeEggs = false  

// 삼항 연산자대신 이렇게 쓸 수 있습니다.

val breakfast =  

if (likeEggs) "계란후라이"  

else "사과"  

println(s"아침으로 ${breakfast}를 먹어요")