Mobile:
+91-8879712614 Phone:022-42669636 | Email : hadoopexam@gmail.com
admin@hadoopexam.com
All Products Spark IBM MapR Hortonworks Cloudera NiFi
Amazon AWS SAS
HBase Cassandra
Salesforce Oracle Cloud & Java Android To Activate
Free
Resources
Forum
Subscribe
Annual
Subscription (50%+49% off) Author/Trainer
For Business
Blog 
25000+ Learners upgraded/switched career Testimonials
All
Certifications
preparation material is for renowned vendors like
Cloudera, MapR, EMC, Databricks,SAS, Datastax, Oracle,
NetApp etc , which has more value, reliability and
consideration in industry other than any training
institutional certifications.
Note
:
You can choose more than one product to have custome
package created from below and send email to
hadoopexam@gmail.com to get discount.Premium
Trainings Courses : HadoopExam focuses
on in depth learning with the hands-on session setting up the
environment than executing solution and doing hands on
that. Below are the available trainings and we are keep adding
new trainings. These trainings is being used and subscribed by
Devloper, Tester, Administrator, Enterprise(to train their team)
and Trainer globally. These trainings are well organized and
step by step solutions to learning, and in lesser time as per
your convenience you can complete these and even re-visit as
required.
Previous | Next | Audio Book for Spark Interview Questions is available here | Top 150 Latest Spark Interview Questions | Quickly go through Spark Training Python & Scala
Question: Please explain the Spark execution model?
Answer: Spark execution model have following concepts
- Driver: An application maps to a single driver process. Driver process manages the job flow and schedule tasks and is available the entire time the application is running. Typically, this driver process is the same as the client process used to initiate the job, although when run on YARN, the driver can run in the cluster. In interactive mode, the shell itself is the driver process.
- Executor: For a single application/driver set of executor processes are distributed across the hosts in a cluster. The executors are responsible for performing work, in the form of tasks, as well as for storing any data that you cache. Executor lifetime depends on whether dynamic allocation is enabled. An executor has a number of slots for running tasks, and will run many concurrently throughout its lifetime.
Figure 1:To-do Update this image
- Stage: A stage is a collection of tasks that run the same code, each on a different subset of the data.
Question: What is Dynamic Allocation?
Answer: Dynamic allocation allows Spark (Only on YARN) to dynamically scale the cluster resources allocated to your application based on the workload. When dynamic allocation is enabled and a Spark application has a backlog of pending tasks, it can request executors. When the application becomes idle, its executors are released and can be acquired by other applications.
When Spark dynamic resource allocation is enabled, all resources are allocated to the first submitted job available causing subsequent applications to be queued up. To allow applications to acquire resources in parallel, allocate resources to pools and run the applications in those pools and enable applications running in pools to be preempted.
Question: How Spark Streaming applications are impacted with Dynamic Allocation?
Answer: When Dynamic Allocation is enabled in Spark, which means that executors are removed when they are idle. However, Dynamic allocation is not effective in case of Spark Streaming. In Spark Streaming, data comes in every batch, and executors will run whenever data is available. If the executor idle timeout is less than the batch duration, executors are constantly added and removed. If executor idle timeout is greater than the batch duration, executors are never removed. Hence, it is recommended that you disable the Dynamic Allocation for Spark streaming by setting spark.dynamicAllocation.enabled to flase.
Question: When you submit Spark streaming application on local mode and not on Hadoop YARN, then it is must to have two threads, why?
Answer: As we have discussed previously, when Spark Streaming application is executed, it require at least two threads, one for receive data and one for processing that data.
Question: How do you enable Fault-tolerant data processing in Spark streaming?
Answer: If the Driver host for a Spark Streaming application fails, it can lose data that had been received but not yet processed. To ensure that no data is lost, you can use Spark Streaming recovery. Spark writes incoming data to HDFS as it is received and uses this data to recover state if a failure occurs.
Previous | Next | Audio Book for Spark Interview Questions is available here | Top 150 Latest Spark Interview Questions | Quickly go through Spark Training Python & Scala
- Training Access: No time constraint and Any future enhancements on same and subscribed training will be free.
- Question Bank
(Online Simulator): Now you can have free
updates for additional or updated Questions till
your subscription is active.
- On
Mobile/Tablet/Desktop : You know this
particular exam you can access from your mobile,
tablet or Desktop. You just need internet access and
browser.
- Training Institute : Do you know many of the training institutes subscribe this products from HadoopExam to train their students.
Do
you know?
Read all testimonials its learners voice : Testimonials
Read all testimonials its learners voice : Testimonials