resourece manager

Cluster vs Client: Execution modes for a Spark application

May 19, 2020August 30, 2022Apache Spark, Big Data and Fast Data, Java, Studio-ScalaBig Data Analytics, resourece manager, spark ecosystem, Spark Job, yarn

Reading Time: 3 minutes Whenever we submit a Spark application to the cluster, the Driver or the Spark App Master should get started. And the Driver will be starting N number of workers. Spark driver will be managing spark context object to share the data and coordinates with the workers and cluster manager across the cluster. Cluster Manager can be Spark Standalone or Hadoop YARN or Mesos. Workers will Continue Reading

Introduction to Mesos

September 23, 2017Studio-DevOps, Studio-Scaladeployment, mesos, resourece manager, yarn

Reading Time: 4 minutes What is Mesos ? In layman’s term, Imagine a busy airport. Airplanes are constantly taking off and landing. There are multiple runways, and an airport dispatcher is assigning time-slots to airplanes to land or takeoff. So Mesos is the airport dispatcher, runways are compute nodes, airplanes are compute tasks, and frameworks like Hadoop, Spark and Google Kubernetes are airlines companies. In technical terms, Apache Mesos Continue Reading