How to use Scala along with Spark for a Java Developer

Below are tips and tricks I have learnt about Spark with Scala

1. How to create dataset of custom objects

   import spark.implicits._ class MyObj(val i: Int) // ... val d = spark.createDataset(Seq(new MyObj(1),new MyObj(2),new MyObj(3)))

   Will give you error, so instead use kyro encoder

   import spark.implicits._ class MyObj(val i: Int) implicit val myObjEncoder = org.apache.spark.sql.Encoders.kryo[MyObj]

   // ...
   

   val d = spark.createDataset(Seq(new MyObj(1),new MyObj(2),new MyObj(3)))

2. 
   
3.