spark ecosystem

Cluster vs Client: Execution modes for a Spark application

May 19, 2020August 30, 2022Apache Spark, Big Data and Fast Data, Java, Studio-ScalaBig Data Analytics, resourece manager, spark ecosystem, Spark Job, yarn

Reading Time: 3 minutes Whenever we submit a Spark application to the cluster, the Driver or the Spark App Master should get started. And the Driver will be starting N number of workers. Spark driver will be managing spark context object to share the data and coordinates with the workers and cluster manager across the cluster. Cluster Manager can be Spark Standalone or Hadoop YARN or Mesos. Workers will Continue Reading

Getting Started with the Apache Spark

May 9, 2017August 8, 2018Apache Spark, Spark, Studio-ScalaApache Spark, installation of spark on linux, introduction to spark, Spark, spark architecture, spark ecosystem, Spark Streaming

Reading Time: 2 minutes Apache Spark is an open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley’s AMPLab, and open sourced in 2010 as an Apache project. Spark has several advantages compared to other big data and Map Reduce technologies like Hadoop and Storm. Apache Spark is an improvement on the original Hadoop MapReduce Continue Reading