Category Archives: Streaming

Introduction to Structured Streaming


Hello!!  Knoldus had organized half an hour session on Structured Streaming briefing about the API changes, how it is different from the early Stream Computation paradigm (DStreams) and example API demonstration. Hope you will enjoy. Below are the slides and Video … Continue reading

Posted in apache spark, Scala, Spark, Streaming | 1 Comment

Twitter’s tweets analysis using Lambda Architecture


Hello Folks, In this blog i will explain  twitter’s tweets analysis with lambda architecture. So first we need to understand  what is lambda architecture,about its component and usage. According to Wikipedia, Lambda architecture is a data processing architecture designed to handle … Continue reading

Posted in Akka, akka-http, Apache Kafka, apache spark, Architecture, Batch, big data, Cassandra, Scala, Spark, Streaming | 5 Comments

Lambda Architecture with Spark


Hello folks, Knoldus  organized a knolx session on the topic : Lambda Architecture with Spark. The presentation covers lambda architecture and implementation with spark.In the presentaion we will discuss components of lambda architecure like batch layer,speed layer and serving layer.We will … Continue reading

Posted in Akka, akka-http, Cassandra, Scala, Spark, Streaming | Tagged | 2 Comments

Meetup: Stream Processing Using Spark & Kafka


Knoldus organized a Meetup on Friday, 9 September 2016. Topics which were covered in this meetup are: Overview of Spark Streaming. Fault-tolerance Semantics & Performance Tuning. Spark Streaming Integration with  Kafka. Meetup code sample available here Real time stream processing … Continue reading

Posted in Apache Kafka, apache spark, Best Practices, big data, Elasticsearch, Scala, Spark, Streaming | 1 Comment

Building Analytics Engine Using Akka, Kafka & ElasticSearch


In this blog , I will share my experience on building scalable, distributed and fault-tolerant  Analytics engine using Scala, Akka, Play, Kafka and ElasticSearch. I would like to take you through the journey of  building an analytics engine which was primarily … Continue reading

Posted in Akka, akka-http, Amazon, Amazon EC2, Apache Kafka, Architecture, AWS, AWS Services, Batch, Best Practices, big data, Cassandra, database, Elasticsearch, Java, Non-Blocking, NoSql, Reactive, S3, Scala, Streaming, Web | 10 Comments

Getting close to Apache Flink, albeit in a Träge manner – 3


In the last two blogs on Flink, I hope to have been able to underline the primacy of Windows in the scheme of things of Apache Flink’s streaming. I have shared my understanding of two types of Windows that can … Continue reading

Posted in Apache Flink, Scala, Streaming | Leave a comment

Getting close to Apache Flink, albeit in a Träge manner – 2


From the preceding post in this series In the last blog , we had taken a look at Flink’s CountWindow feature. Here’s a quick recap: As a stream of events enter a Flink-based application, we can apply a transformation of … Continue reading

Posted in Flink, Scala, Streaming | 2 Comments

Getting close to Apache Flink, albeit in a Träge manner – 1


Of late, I have begun to read about Apache Flink. Apache Flink (just Flink hereafter), is an ‘open source platform for distributed stream and batch data processing’, to quote from the homepage.  What has caught my interest is Flink’s idea … Continue reading

Posted in Flink, Scala, software, Streaming | 1 Comment