Apache Spark

Spark: Why should we use SparkSession ???

Apache Spark, Scala, Spark, Tech Blogs

Spark 2.0 is the next major release of Apache Spark. This brings major change for the level of abstraction for the spark API and libraries. The release has the major change for the ones who want to make use of all the advancement in this release, So in this blog post, I’ll be discussing Spark-Session. Need Of Spark-Session

Spark vs MapReduce: Which is better?

Apache Spark

Both the technologies are equipped with amazing features, however with the increased need for real-time analytics, these two giving tough competition to each other What are MapReduce and Spark? MapReduce:- MapReduce is a programming model for processing huge amounts of data in a parallel and distributed. In this model, there are two tasks that are undertaken Map and Reduce and there is a map function Continue Reading

Spark Structured Streaming with Elasticsearch

Apache Spark, Big Data and Fast Data, Elasticsearch, Scala, Streaming, Streaming Solutions, , , ,

There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading

kafka with spark

RDD: Spark’s Fault Tolerant In-Memory weapon

Apache Spark, Big Data and Fast Data, Scala, Tech Blogs, , , ,

A fault-tolerant collection of elements that can be operated on in parallel:  “Resilient Distributed Dataset” a.k.a. RDD RDD (Resilient Distributed Dataset) is the fundamental data structure of Apache Spark which are an immutable collection of objects which computes on the different node of the cluster. Each and every dataset in Spark RDD is logically partitioned across many servers so that they can be computed on Continue Reading

kafka with spark

Spark Unconstructed | Deep dive into DAG

Apache Spark, cluster, Java, Scala, Tech Blogs, ,

Apache Spark is all the rage these days. People who work with Big Data, Spark is a household name for them. We have been using it for quite some time now. So we already know that Spark is lightning-fast cluster computing technology, it is faster than Hadoop MapReduce. If you ask any of these Spark techies, how Spark is fast, they would give you a Continue Reading

CuriosityX: RDDs – The backbone of Apache Spark

Apache Spark, HDFS, Java, Scala, Spark, , , , , ,

In our last blog, we tried to understand about using the spark streaming to transform and transport data between Kafka topics. After reading that many of the readers asked us to give a brief description of RDDs in Spark which we used. So, this blog is totally dedicated to the RDDs in Spark. So let’s start with the very basic question that comes to our mind Continue Reading

Code Combat II : The Code Battle For The Vanguard Continues…

Akka, Apache Kafka, Apache Spark, Blockchain, Java, ML, AI and Data Engineering, Scala, , , , , , ,

“If you can dream it, you can do it. ”  -Walt Disney For some coding is a job. For some, it is an exercise. But for us folks here at Knoldus, it’s a Passion. So in order to bring a twist in the daily work schedule, Knoldus held an overnight Hackathon competition within the organization on 18th May 2018 which presented an opportunity for every Knolder(employees Continue Reading

Knoldus-Spark-AI-Summit-2018

A quick recap of SPARK + AI SUMMIT 2018

Apache Spark, event, Scala

Spark + AI Summit was the world’s largest big data event focused entirely on Apache Spark assembling 4,000 people(best engineers, scientists, analysts, and executives) from over 40 countries attending to share their knowledge and receive expert training on this open-source powerhouse. If you didn’t make it, don’t worry; We’ve got you covered! Vikas Hazrati and Ram Indukuri from Knoldus Inc. attended the Spark+AI Summit 2018, Continue Reading

Spark Stream-Stream Join

Tuning spark on yarn

Apache Spark, Scala, Spark

In this blog we will learn how to tuning yarn with spark in both mode yarn-client and yarn-cluster,the only requirement to get started is that you must have a hadoop based yarn-spark cluster with you. In case you want to create a cluster you can follow this blog here. 1. yarn-client mode:  In client mode, the driver runs in the client process, and the application master is only used Continue Reading

Structured Streaming: Philosophy behind it

Apache Spark, Big Data and Fast Data, Scala, Spark, Streaming,

In our previous blogs: Structured Streaming: What is it? & Structured Streaming: How it works? We got to know 2 major points about Structured Streaming – It is a fast, scalable, fault-tolerant, end-to-end, exactly-once stream processing API that helps users in building streaming applications. It treats the live data stream as a table that is being continuously appended/updated which allows us to express our streaming computation as Continue Reading

Structured Streaming: How it works?

Apache Spark, Scala, Spark, Streaming

In our previous blog post – Structured Streaming: What is it? we got to know that Structured Streaming is a fast, scalable, fault-tolerant, end-to-end, exactly-once stream processing API that helps users in building streaming applications. Now it’s time to learn  – How it works? So, in this blog post, we will look at the working of a structured stream via an example. So, let’s take a Continue Reading

Running Spark on DC/OS

Apache Spark, cluster, DevOps, Scala, Spark, ,

Devops engineers for long needed an open source tool to make it easy to deploy the code developed through all the ups and downs to reach this far and is considerably more capable of evolving (pun intended). As we all know in this world of agile we need to shift our requirements after a short duration of time. Be it addition of a feature or tweaking Continue Reading

Structured Streaming: What is it?

Apache Flink, Apache Kafka, Apache Spark, Flink, Java, Scala, Spark, Streaming, , ,

With the advent of streaming frameworks like Spark Streaming, Flink, Storm etc. developers stopped worrying about issues related to a streaming application, like – Fault Tolerance, i.e., zero data loss, Real-time processing of data, etc. and started focussing only on solving business challenges. The reason is, the frameworks (the ones mentioned above) provided inbuilt support for all of them. For example: In Spark Streaming, by just adding Continue Reading

%d bloggers like this: