Apache Spark

Start/Deploy Apache Spark application programmatically using spark launcher

Reading Time: 1 minute Sometimes we need to start our spark application from the another scala/java application. So we can use SparkLauncher. we have an example in which we make spark application and run it with another scala application. Let see our spark application code. import org.apache.spark.SparkConf import org.apache.spark.SparkContext object SparkApp extends App{ val conf=new SparkConf().setMaster(“local[*]”).setAppName(“spark-app”) val sc=new SparkContext(conf) val rdd=sc.parallelize(Array(2,3,2,1)) rdd.saveAsTextFile(“result”) sc.stop() } This is our simple spark Continue Reading

Stateful transformation on Dstream in apache spark with example of wordcount

Reading Time: 2 minutes Sometimes we have a use-case in which we need to maintain state of paired Dstream to use it in next Dstream . So we are taking a example of stateful wordcount in socketTextStreaming. Like in wordcount example if word “xyz” comes twice is in first Dstream or window, it reduce it and its value is 2 but its state will lost in the next Dstream Continue Reading

Shufflling and repartitioning of RDD’s in apache spark

Reading Time: 3 minutes To write the optimize spark application you should carefully use transformation and actions, if you use wrong transformation and action will make your application  slow. So when you are writing application some points should be remember to make your application more optimize. 1. Number of partitions when creating RDD By default spark create one partition for each block of the file in HDFS it is Continue Reading

Setup a Apache Spark cluster in your single standalone machine

Reading Time: 2 minutes If we want to make a cluster in standalone machine we need to setup some configuration. We will be using the launch scripts that are provided by Spark, but first of all there are a couple of configurations we need to set first of all setup a spark environment so open the following file or create if its not available with the help of template Continue Reading

Knoldus Pune Careers - Hiring Freshers

Get a head start on your career at Knoldus. Join us!