Answer: Databricks has changed a lot in their new release of Apache Spark certification. Below two are the major changes in the exam

• There would be separate exam for Spark Scala and Spark Python. In future same is expected to be available in Java and R language.
• Databricks have upgraded certification and will be testing on Spark 2.x platform of Apache Spark.

Answer: You should select certification based on your programming language skill. If you are from Java/Scala background then go for Scala based Spark certification. And if you know Python programming language than go for Python Spark certification.

With regards to performance: that was the case on older version of Spark where Scala Spark was better performant that PySpark. But in Spark 2.x this is not the case (Because of their Optimizer), whatever programming language you use either Java/Scala/Python/R all are same on performance. Only exception to this is User Defined Function.

Answer: You will be asked question on following topics

• Basic concepts of Apache Spark
• Spark Structured Streaming (Concepts + Programming questions)
• Architecture of Apache Spark which include following topics
  • Driver Program
  • Cluster Manager
  • Client Mode vs Cluster Mode
  • Executors and Tasks
  • SparkSession
• Spark Performance optimization and debugging the performance issues.
  • You must understand stages, tasks etc. concepts
  • You should be able to check where to use cache
  • What is the use of checkpointing
• Spark SQL
  • You should be well versed to writing SQL queries using Spark SQL Platform
  • You should be able to use DataFrame/Dataset API
  • You should be able to select correct queries or code snippet for expected result
  • You should be able to select correct output based on the given code snippet
• GraphFrame:
  • You should be able to answer various graph problems.
  • Lot of questions are already included in https://www.hadoopexam.com certification simulator
• Deploying applications in Cluster

Answer: Syllabus mentioned for the Databricks Spark certification is very abstract. And it is not given in detail what they will be asking in the exam. And we expect quite a good amount of questions based on the RDD and programming question will be asked for various RDD API. Reason RDD is still in focus, is because whether your use Spark1.x or Spark 2.x their underline processing engine works on RDD only. Hence, concepts of RDD must be cleared. And you wanted to apply some custom optimization, wanted to do performance tuning etc. then you should know how RDD works. Even if you are using distributed shared variables like Broadcast variable and Accumulators then you will be using RDD.

You can convert your DataFrame to RDD, as well as RDD to DataFrame with simple API. Hence, you must have a good experience with the Spark RDD programming as well. And it is expected you know while working with the Spark.

Answer: As we have mentioned earlier they can ask questions based on whatever programming language you have chosen for certification. So before purchasing the certification make sure what programming language you are using

• Python
• Scala

Answer:  As of now there is not a single book available which focus on these Spark certifications. We highly recommend that you use below study material for the preparing respective certifications. These material is regularly updated and new things are added.

Spark Scala

• Apache Spark Professional (Include 2.x latest Version setup) Training with Hands On Lab
• Spark 2.X SQL (Using Scala)  Professional Training with Hands On Sessions
• Databricks Certified Developer Apache Spark 2.x for Scala (360+ Questions and answers)

Python Spark (PySpark)

• PySpark 2.X (Using Python)  Professional Training with Hands On Sessions
• Databricks Certified Developer Apache Spark 2.x for Python 

Answer: From the Apache it is one of the most actively worked framework. In recent years BigData, Real time Data processing, Artificial Intelligence and many other things pushed high. And all this need a processing engine which can process such things efficiently. Even Hadoop Mapreduce which become suddenly popular, is replaced by Spark computation engine. There are almost more than 1000 contributors on the open source platform.

After Spark 2.0, it is very easy to learn. Its API is very intuitive as well if you are good at SQL queries then it make it much easier for you to learn. If you are a programmer than DataFrame/Dataset API would help you a lot for working with the Spark.

There are many organizations have pushed Spark applications in the production. Which proves the quality and reliability of the Spark framework.

Companies already having Hadoop cluster do not have to create separate Spark cluster. They can use their existing framework for the running Spark jobs on the same cluster. Whether it is written using Java/Scala/Python or R language.

Always having new technologies knowledge will give you the opportunity to draw more salary. And less chance of job loss. You can switch your career and Spark is one of them for sure.

Answer:  There is a myth in IT industry that certification does not help in career. This is not at all true. Having certification certainly helps in following ways

• You will know all the hidden features of a technology. If you go for certification
• It shows your career focus
• While resume shortlisting it is given priority (Because first shortlisting is done by recruitment team, they don’t have enough knowledge about technology. Hence, they look for your credentials in the resume).
• First impression on the interviewer.
• Interviewer will focus on things which you have written in resume.
• You will categorized in separate category.
• It will give confidence during the interview and while working in the organization.
• So avoid all the people who have –ve thinking about learning. Learning can never be costly and time wasting (universal truth).
• Certainly and additional feather in your hat.
• There are many other latent benefits for doing certification.

Answer: There are various other vendors who conduct certification on Spark as below

• Cloudera Hadoop and Spark certification
• Hortonworks Spark certification
• MapR Spark Scala Certification

Answer: No we don’t give priority to any vendor. It varies based on many factors.

• Like if you wanted to get certified in both Hadoop and Spark then go for Cloudera Hadoop and Spark certification. And you have to have knowledge how to use Cloudera platform.
• If you are working on MapR platform then you can go for MapR Spark certification. Even other advantage is that MapR Spark certification is not as lengthy as Databricks Spark certification. You can prepare for MapR Spark certification quite less time. You can see pross and cons that Databricks is more involved with the Spark and really tough one among the all Spark certification.
• Hortonworks Spark certification: This is again Hands on certification for the Spark. And have limited syllabus and specific objectives are given.

Answer: It is very tricky question to answer. We recommend learn both the programming language. These are beautiful language to work upon. But based on the following career path you can choose respective programming language.

Scala:

• Java programmer should go for this
• If you want to become Data Engineer than go for this
• If you want to work on Data Cleaning and collecting Data than go for this.
• If you already know Java/Scala than go for this

Python:

• If you know Python than go for PySpark.
• If you are on Business Analytics profile go for PySpark
• I want to become Data Scientist, you can use either PySpark or Scala Spark

It should not be considered based on the fact that Spark is written in Scala, so I should give preference to Spark Scala. Not at all true after Spark 2.x version.

Answer: These are the common area and you must keep in mind

• You will not get too many questions from RDD programming but for sure 2 to 4 questions you will be getting on RDD.
• You must know the partitioning and shuffling, how to avoid shuffling. What is narrow transformation? And what is wide transformation? Expect 3-4 questions from this section as well.
• You must know the impact of User Defined function when you write in Python and Scala. How performance does is impacted. What must be taken care for the UDAF (User Defined Aggregate function etc).
• We have detailed chapter on how to create UDF and UDAF, sure shot 3-4 questions on this section. This SparkSQL training would help for this.
• How to read data formats like Parquet, ORC, CSV, JSON, XML and AVRO. You must know various options for reading the data using DataFrameReader object. What all options are available for reading such data?
• Once you read and process the data, how would save this data like in HDFS, S3, local file system. There are various options and syntax, you must know them.
• For reading and writing assume you will get 6-8 questions in total.
• Sure shot 2 question from GraphFrame. In this (Scala and Python Spark) simulator we have covered very well for the GraphFrame questions.
• Spark Machine Learning Library question, you will get around 2 question from this section.
• Major updates have been done for following two components in Spark 2.x
  • Project Tungsten
  • Catalyst Optimizer
• You must know these two section in detail 4-6 question is expected on that.
• What is the use of Encoder (Sterilization and Deserialization) , learn this to answer Spark question.