Big Data and Fast Data

Want to expose an API? Reactive Options

In the modern digital world, a system which cannot talk to other systems or does not allow other systems to talk to itself is bound to fail. A lot of organizations that we work with be it large banks, healthcare institutions, travel booking and predictions, industrial analysers, would like to expose their ecosystem through an API so that it is available as a platform for other companies and products to consume and innovate on.

Knolx: How Spark does it internally?

Knoldus has organized a 30 min session on Oct 12 at 3:30 PM. The topic was How Spark does it internally? Many people have joined and enjoyed the session. I am going to share the slides and the video here. Please let me know if you have any question related to linked slides.   How Spark Does It Internally? from Knoldus Inc.   Here’s the video of the Continue Reading

Is Apache Flink the future of Real-time Streaming?

In our last blog, we had a discussion about the latest version of Spark i.e 2.4 and the new features that it has come up with. While trying to come up with various approaches to improve our performance, we got the chance to explore one of the major contenders in the race, Apache Flink. Apache Flink is an open source platform which is a streaming Continue Reading

kafka with spark

Apache Spark 2.4: Adding a little more Spark to your code

Continuing with the objectives to make Spark faster, easier, and smarter, Apache Spark recently released its fifth release in the 2.x version line i.e Spark 2.4. We were lucky enough to experiment with it so soon in one of our projects. Today we will try to highlight the major changes in this version that we explored as well as experienced in our project. In our Continue Reading

CAP Theorem for the distributed systems

A few days back I completed the certification for the 1st course of the Lightbend Reactive Architecture Advanced i.e. Building Scalable Systems. I found this course very helpful and informative to get the idea of Reactive architecture. So if you have not started yet, please go there and lets become reactive. There are few foundational courses as well to build the foundation of reactive architecture. Continue Reading

Scheduling Jobs with Akka Scheduler

Hey folks, in this blog I am going to explain how can you schedule jobs that you want to repeat over a certain period of time with the help of Akka Scheduler. Suppose you have a use-case in which you want some cleaning background process to run a cleanup-repository method to delete records after a fixed interval of time, then look nowhere else because Akka scheduler Continue Reading

Still with Spring? 9 reasons to Akka

As a niche consulting and development organization we end up in a lot of enterprises who would like to modernize, would like to build web-scale products but they are not ready to look beyond Spring. It is a strongly debated topic and there are reasons for still going the Spring way but the reasons are less and less as we go ahead. For starters, this Continue Reading

kafka with spark

Tuning a Spark Application

Having trouble optimizing your Spark application? If yes, then this blog will surely guide you on how you can optimize it and what parameters should be tuned so that our spark application gives the best performance. Spark applications can cause a bottleneck due to resources such as CPU, memory, network etc. We need to tune our memory usage, data structures tuning, how RDDs need to Continue Reading

HDFS: A Conceptual View

There has been a significant boom in distributed computing over the past few years. Various components communicate with each other over network inspite of being deployed on different physical machines. A distributed file system (DFS) is a file system with data stored on a server. The data is accessed and processed as if it was stored on the local client machine. The DFS makes it convenient to share information Continue Reading

Alpakka – Connecting Kafka and ElasticSearch to Akka streams

In our previous blog, we had a look at what Akka streams are and how they are different from the other streaming mechanisms we have. In this blog, we will be taking a little step forward into the world of Akka Streams. In order to work with Akka streams, we need a mechanism to connect Akka Streams to the existing system components. That is where Alpakka Continue Reading

Akka Streams: Is it a Solution to Your Streaming Problems?

A few days earlier, in our project, we were using Spark streaming and initially, it worked like a charm. But as we were very close to completion of our use case, the unexpected occurred. Spark does have a lot of interesting features, but we had some more custom needs such as running a ton of varying jobs with different actors/flows. Also, we needed something which Continue Reading

Welcome, Akka Typed !!

While I was attending ScalaDays Berlin 2018, I experienced many great ideas but the one which intrigued me the most was “Farewell Any => Unit, welcome Akka Typed!” by Heiko Seeberger where he explained about Akka Typed APIs. In this blog I am going to explain what I learned until now about Akka Typed, therefore, I named this blog “Welcome, Akka Typed !!”, so let’s Continue Reading

Knoldus Pune Careers - Hiring Freshers

Get a head start on your career at Knoldus. Join us!