Author Archives: Mahesh Chand Kandpal

About Mahesh Chand Kandpal

Explorer + Technology Enthusiast + Foodie + Movie Buff

Having Issue How To Order Streamed Dataframe ?


A few days ago, i have to perform aggregation on streaming dataframe. And the moment, i apply groupBy for aggregation, data gets shuffled. Now the situation arises how to maintain order? Yes, i can use orderBy with streaming dataframe using … Continue reading

Posted in Apache Kafka, apache spark, big data, Scala, Spark, Streaming | Tagged , , , , , , , , , , | Leave a comment

Unit Testing Of Kafka


Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Generally, data is published to topic via Producer API … Continue reading

Posted in Apache Kafka, Scala, scalatest, Streaming, testing | Tagged , , , , | 2 Comments

Exploring Spark Structured Streaming


Hello Spark Enthusiasts, Streaming apps are growing more complex. And it is getting difficult to do with current distributed streaming engines. Why streaming is hard ? Streaming computations don’t run in isolation. Data arriving out of time order is a … Continue reading

Posted in apache spark, Scala, Streaming | Tagged , | Leave a comment

Spark Streaming vs Kafka Stream


The demand for stream processing is increasing a lot these days. The reason is that often processing big volumes of data is not enough. Data has to be processed fast, so that a firm can react to changing business conditions … Continue reading

Posted in Apache Kafka, apache spark, big data, Scala, Streaming | Tagged , | 1 Comment

Jump Higher With Trampoline


In our earlier blog The Tale of ‘Tail Recursion’, we have talked about tail recursion. And In this blog, we will be talking about limitation of tail recursion and its solution in scala. Unfortunately JVM doesn’t support tail call optimization, … Continue reading

Posted in Scala | Tagged , , , , , | 1 Comment

The Tale of ‘Tail Recursion’


Recursion in computer science is a method where the solution to a problem depends on solutions to smaller instances of the same problem (as opposed to iteration). Recursions are really cool and they are highly expressive. For example consider factorial … Continue reading

Posted in Scala | Tagged , , | 4 Comments