In-memory computing

Writing Unit Test for Apache Spark using Memory Streams

March 24, 2021April 19, 2021Apache Spark, scalatest, Studio-Scala#scalatest, Apache Spark, In-memory computing, Streaming Spark, Unit testing Spark Streaming

Reading Time: 2 minutes In this post, we are going to look into how we can leverage apache spark’s memory streams for Unit testing What is it ? Apache spark’s memory streams is a concrete streaming source of memory data source that supports reading in Micro-Batch Stream Processing. Lets jump into it We will be using a memory stream writing some test data in memory as a stream. We Continue Reading

Sharing RDD’s states across Spark applications with Apache Ignite

July 26, 2016September 10, 2018Apache Spark, Spark, Studio-ScalaApache Ignite, Apache Spark, In-memory computing, Shared RDD, Spark

Reading Time: 4 minutes Apache Ignite offers an abstraction over native Spark RDDs such that the state of RDDs can be shared across spark jobs, workers and applications which is not possible with native Spark RDDS. In this blog, we will walk through the steps on how to share RDDs between two spark Application. Preparing Ingredients To test the Apache Ignite with Apache Spark application we need at least one master Continue Reading