Category Archives: Streaming

Rules while working with stream in Java 8

First, let’s have a basic understanding of stream. Then we will have a look at the side effects that can occur while working with streams. Stream represents a sequence of objects from a source, which supports aggregate operations. One thing … Continue reading

Posted in Java, Best Practices, Streaming | Tagged , , , , , , , , | Leave a comment

One-way & two-way streaming in a Lagom application

Now a days streaming word is a buzz word and you should have heard many types of streaming till now i.e. kafka streaming, spark streaming etc etc. But in this blog we will see a new type of streaming i.e … Continue reading

Posted in Akka, Best Practices, big data, Functional Programming, github, Java, knoldus, Messages, Reactive, Scala, Streaming, Web Services | Leave a comment

Apache Storm: Architecture

Apache Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing the realtime computation. Storm is simple, can be used … Continue reading

Posted in big data, Clojure, Scala, Streaming | 1 Comment

Case Study to understand Kafka Consumer and its offsets

In this blog post, we will discuss mainly Kafka Consumer and its Offsets. We will understand this using a case study implemented in Scala. This blog post assumes that you are aware of basic Kafka terminology. CASE STUDY: The Producer … Continue reading

Posted in Apache Kafka, big data, Functional Programming, knoldus, Scala, Streaming | 3 Comments

What’s new in Apache Spark 2.2

Apache recently released a newer version of Spark i.e Apache Spark2.2. The new version comes with new improvements as well as the addition of new functionalities. The major addition to this release is Structured Streaming. It has been marked as production … Continue reading

Posted in apache spark, big data, Scala, Spark, Streaming | Tagged , , , , , , , , , , | 3 Comments

Basic Example for Spark Structured Streaming & Kafka Integration

The Spark Streaming integration for Kafka 0.10 is similar in design to the 0.8 Direct Stream approach. It provides simple parallelism, 1:1 correspondence between Kafka partitions and Spark partitions, and access to offsets and metadata. However, because the newer integration … Continue reading

Posted in Scala, Spark, Streaming | Tagged , | 8 Comments

Having Issue How To Order Streamed Dataframe ?

A few days ago, i have to perform aggregation on streaming dataframe. And the moment, i apply groupBy for aggregation, data gets shuffled. Now the situation arises how to maintain order? Yes, i can use orderBy with streaming dataframe using … Continue reading

Posted in Apache Kafka, apache spark, big data, Scala, Spark, Streaming | Tagged , , , , , , , , , , | 1 Comment

Self-Learning Kafka Streams with Scala – #1

A few days ago, I came across a situation where I wanted to do a stateful operation on the streaming data. So, I started finding possible solutions for it. I came across many solutions which were using different technologies like … Continue reading

Posted in Apache Kafka, Scala, Streaming | Tagged | 2 Comments

Unit Testing Of Kafka

Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Generally, data is published to topic via Producer API … Continue reading

Posted in Apache Kafka, Scala, scalatest, Streaming, testing | Tagged , , , , | 3 Comments

Meetup: Stream processing using Kafka

Knoldus organized a Meetup on Friday, 7th April 2017 at 4:00 PM which was presented by Himani Arora and me(Prabhat Kashyap). Topics which were covered in this meetup: What is Stream processing Advantages of stream processing Type of stream processing What are KStreams Use cases … Continue reading

Posted in Apache Kafka, Scala, Streaming | Tagged , , | Leave a comment