Why Spark technology in so much news?
Answer: From the Apache it is one of the most actively worked framework. In recent years BigData, Real time Data processing, Artificial Intelligence and many other things pushed high. And all this need a processing engine which can process such things efficiently. Even Hadoop MaPreduce which become suddenly popular, is replaced by Spark computation engine. There are almost more than 1000 contributors on the open source platform.
After Spark 2.0, it is very easy to learn. Its API is very intuitive as well if you are good at SQL queries then API/SQL makes it much easier for you to learn. If you are a programmer than DataFrame/Dataset API would help you a lot for working with the Spark.
There are many organizations who had pushed Spark applications in the production. Which proves the quality and reliability of the Spark framework.
Companies already having Hadoop cluster do not have to create separate Spark cluster. They can use their existing framework for the running Spark jobs on the same cluster. Whether it is written using Java/Scala/Python or R language.
Always having new technologies knowledge will give you the opportunity to draw more salary. And less chance of job loss. You can switch your career and Spark is one of them for sure.