Streaming Solutions

Alpakka – Connecting Kafka and ElasticSearch to Akka streams

In our previous blog, we had a look at what Akka streams are and how they are different from the other streaming mechanisms we have. In this blog, we will be taking a little step forward into the world of Akka Streams. In order to work with Akka streams, we need a mechanism to connect Akka Streams to the existing system components. That is where Alpakka Continue Reading

Akka Streams: Is it a Solution to Your Streaming Problems?

A few days earlier, in our project, we were using Spark streaming and initially, it worked like a charm. But as we were very close to completion of our use case, the unexpected occurred. Spark does have a lot of interesting features, but we had some more custom needs such as running a ton of varying jobs with different actors/flows. Also, we needed something which Continue Reading

Spark Structured Streaming with Elasticsearch

There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading

Structured Streaming: Philosophy behind it

In our previous blogs: Structured Streaming: What is it? & Structured Streaming: How it works? We got to know 2 major points about Structured Streaming – It is a fast, scalable, fault-tolerant, end-to-end, exactly-once stream processing API that helps users in building streaming applications. It treats the live data stream as a table that is being continuously appended/updated which allows us to express our streaming computation as Continue Reading

Structured Streaming: How it works?

In our previous blog post – Structured Streaming: What is it? we got to know that Structured Streaming is a fast, scalable, fault-tolerant, end-to-end, exactly-once stream processing API that helps users in building streaming applications. Now it’s time to learn  – How it works? So, in this blog post, we will look at the working of a structured stream via an example. So, let’s take a Continue Reading

Structured Streaming: What is it?

With the advent of streaming frameworks like Spark Streaming, Flink, Storm etc. developers stopped worrying about issues related to a streaming application, like – Fault Tolerance, i.e., zero data loss, Real-time processing of data, etc. and started focussing only on solving business challenges. The reason is, the frameworks (the ones mentioned above) provided inbuilt support for all of them. For example: In Spark Streaming, by just adding Continue Reading

Kafka And Spark Streams: The happily ever after !!

Hi everyone, Today we are going to understand a bit about using the spark streaming to transform and transport data between Kafka topics. The demand for stream processing is increasing every day. The reason is that often, processing big volumes of data is not enough. We need real-time processing of data especially when we need to handle continuously increasing volumes of data and also need Continue Reading

KnolX: Understanding Spark Structured Streaming

Hello everyone, Knoldus organized a session on 05th January 2018. The topic was “Understanding Spark Structured Streaming”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides:

Knolx: Guaranteed No Stress Baby Steps Using Akka Streams Part-II

Hello everyone, Knoldus organized a session on 25th November 2017. The topic was “Guaranteed No Stress Baby Steps Using Akka Streams Part-II”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides:

Knolx: Guaranteed No Stress Baby Steps Using Akka Streams Part-I

Hello everyone, Knoldus organized a session on 28th October 2017. The topic was “Guaranteed No Stress Baby Steps Using Akka Streams Part-I”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides:

Rules while working with stream in Java 8

First, let’s have a basic understanding of stream. Then we will have a look at the side effects that can occur while working with streams. Streams represent a sequence of objects from a source, which supports aggregate operations. One thing to be notified while working with streams  is that, aggregate operation (intermediate operations) are lazy evaluated i.e. they do not start processing the content of Continue Reading

One-way & two-way streaming in a Lagom application

Now a days streaming word is a buzz word and you should have heard many types of streaming till now i.e. kafka streaming, spark streaming etc etc. But in this blog we will see a new type of streaming i.e Lagom-streaming. Lagom-streaming internally uses Akka streams, with the help of which we will see one way & two way streaming. But before going forward, it Continue Reading

Apache Storm: Architecture

Apache Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing the realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use! Components of a Storm cluster Apache Storm cluster Continue Reading

%d bloggers like this: