Spark Interview Questions-3

Question 11: Is it possible to have multiple SparkContext in single JVM?

Answer: Yes, spark.driver.allowMultipleContexts is true (default: false ). If true Spark logs warnings instead of throwing exceptions when multiple SparkContexts are active, i.e. multiple SparkContext are running in this JVM. When creating an instance of SparkContex.

 

Question 12: Can RDD be shared between SparkContexts?

Answer: No, When an RDD is created; it belongs to and is completely owned by the Spark context it originated from. RDDs can’t be shared between SparkContexts.

 

Question 13: In Spark-Shell, which all contexts are available by default?

Answer: SparkContext and SQLContext

 

Question 14: Give few examples, how RDD can be created using SparkContext

Answer: SparkContext allows you to create many different RDDs from input sources like:

  • Scala’s collections: i.e. sc.parallelize(0 to 100)
  • Local or remote filesystems : sc.textFile("README.md")
  • Any Hadoop InputSource : using sc.newAPIHadoopFile

 

Question 15: How would you brodcast, collection of values over the Sperk executors?

Answer: sc.broadcast("hello")

Details
Category: Spark