Spark Structured Streaming with Elasticsearch

September 25, 2018Apache Spark, Big Data and Fast Data, Elasticsearch, Scala, Streaming, Streaming SolutionsApache Spark, elasticsearch, Elasticsearch with structured streaming, Streaming, Structured Streaming

There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading

CuriosityX: RDDs – The backbone of Apache Spark

July 30, 2018July 31, 2018Apache Spark, HDFS, Java, Scala, SparkApache Spark, Hadoop, java, partitioning in apache spark, RDD, scala, Spark

In our last blog, we tried to understand about using the spark streaming to transform and transport data between Kafka topics. After reading that many of the readers asked us to give a brief description of RDDs in Spark which we used. So, this blog is totally dedicated to the RDDs in Spark. So let’s start with the very basic question that comes to our mind Continue Reading

They said Spark Streaming simply means Discretized Stream

March 23, 2018March 23, 2018Apache Spark, Scala, SparkApache Spark, scala, Spark, spark 2.0, technology

I am working in a company (Knoldus Software LLP) where Apache Spark is literally running into people’s blood means there are certain people who are really good at it. If you ever visit our blogging page and search for stuff related to spark, you will find enough content which is capable of solving your most of spark related queries, starting form introductions to solutions for Continue Reading

Difference between Apache Hadoop and Apache Spark Mapreduce

December 1, 2017August 9, 2018Apache Spark, Big Data and Fast Data, Scalaapache hadoop, Apache Spark, Big Data, Hadoop, scala, Spark

The term Big Data has created a lot of hype already in the business world. Hadoop and Spark are both Big Data frameworks – they provide some of the most popular tools used to carry out common Big Data-related tasks. In this blog, we will cover what is the difference between Apache Hadoop and Apache Spark MapReduce. Introduction Spark – It is an open source Continue Reading

Assimilation of Spark Streaming With Kafka

November 27, 2017August 8, 2018Apache Kafka, Apache Spark, ScalaApache Kafka, Apache Spark, scala, Spark Streaming

As we know Spark is used at a wide range of organizations to process large datasets. It seems like spark becoming main stream. In this blog we will talk about Assimilation of Spark Streaming With Kafka. So, lets get started. How Kafka can be integrated with Spark? Kafka provides a messaging and integration platform for Spark streaming. Kafka act as the central hub for real-time streams of Continue Reading

What’s new in Apache Spark 2.2

October 4, 2017August 9, 2018Apache Spark, Big Data and Fast Data, Scala, Spark, Streamingapache, Apache Spark, Big Data, opensource, scala, Spark, Spark SQL, Spark Streaming, Spark Structured Streaming, spark2.2, technology

Apache recently released a newer version of Spark i.e Apache Spark 2.2. The new version comes with new improvements as well as the addition of new functionalities. The major addition to this release is Structured Streaming. It has been marked as production ready and its experimental tag has been removed. Some of the high-level changes and improvements : Production ready Structured Streaming Expanding SQL functionalities New Continue Reading

Spark Structured Streaming: A Simple Definition

June 18, 2017July 20, 2018Scala, Spark, StreamingApache Spark, Big Data, continuous application, spark 2.0, Structured Streaming

“Structured Streaming”, nowadays we are hearing this term in Apache Spark ecosystem quite a lot, as it is being preached as next big thing in scalable big data world. Although, we all know that Structured Streaming means a stream having structured data in it, but very few of us knows what exactly it is and where we can use it. So, in this blog post Continue Reading

Getting Started with the Apache Spark

May 9, 2017August 8, 2018Apache Spark, Scala, SparkApache Spark, installation of spark on linux, introduction to spark, Spark, spark architecture, spark ecosystem, Spark Streaming

Apache Spark is an open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley’s AMPLab, and open sourced in 2010 as an Apache project. Spark has several advantages compared to other big data and Map Reduce technologies like Hadoop and Storm. Apache Spark is an improvement on the original Hadoop MapReduce Continue Reading

The Dominant APIs of Spark: Datasets, DataFrames and RDDs

April 16, 2017April 17, 2017SparkApache Spark, dataframes, datasets, performance optimization, RDD, space optimization, spark apis

While working with Spark often we come across the three APIs: DataFrames, Datasets and RDDs. In this blog I will discuss the three in terms of use case, performance and optimization. It is essential to keep in mind that there is seamless transformation available between the three DataFrames, Datasets and RDDs. Implicitly the RDD forms the apex of both DataFrame and Datasets. The inception of Continue Reading

Partition-Aware Data Loading in Spark SQL

February 9, 2017July 20, 2018Scala, SparkApache Spark, Best Practices, Big Data, Spark SQL

Data loading, in Spark SQL, means loading data in memory/cache of Spark worker nodes. For which we use to write following code: val connectionProperties = new Properties() connectionProperties.put(“user”, “username”) connectionProperties.put(“password”, “password”) val jdbcDF = spark.read .jdbc(“jdbc:postgresql:dbserver”, “schema.table”, connectionProperties) In here we are using jdbc function of DataFrameReader API of Spark SQL to load the data from table into Spark Executor’s memory, no matter how many rows are Continue Reading

Migration From Spark 1.x to Spark 2.x

December 30, 2016December 31, 2016ScalaApache Spark

Hello Folks, As we know that we have latest release of Spark 2.0, with to much enhancement and new features. If you are using Spark 1.x and now you want to move your application with Spark 2.0 that time you have to take care for some changes which happened in the API. In this blog we are going to get an overview of common changes: Continue Reading

Spark – LDA : A Complete example of clustering algorithm for topic discovery.

October 8, 2016October 12, 2016Apache Spark, Scala, SparkApache Spark, cleaning data, clustering, clustering in distributed systems, distributed LDA, document Clustering, full example, introduction to machine learning, LDA algorithm, lemmatization, Machine Learning, Machine Learning Pipeline, Natural language processing, NLP, opensource, Spark, Spark package, stanford nlp, Stemming, Stopwords removal, Testing, tokenization, Topic identification

In this blog we will be demonstrating the functionality of applying the full ML pipeline over a set of documents which in this case we are using 10 books from the internet. So lets start with first thing first.. What is Clustering ? Clustering is the task of grouping a set of objects in such a way that objects in the same group (called a Continue Reading

Spark – IoT : Combining Big Data Analysis with IoT

October 1, 2016October 1, 2016Apache Spark, IOT, Scala, SparkApache Spark, Big Data, Big Data Analytics, community, IOT, overview, scala, scala-iot, Smart device, Spark, spark iot, web series

Welcome back , folks ! Time for some new gig ! I think that last series i.e. Scala – IOT was pretty amazing , which got an overwhelming response from you all which resulted in pumping up the idea of this new web-series Spark-IOT. So let’s get started, What was the motivation ? I have been active in the IoT community here, and I found Continue Reading

Apache Spark