Knoldus Blogs

Windows operator: Heart of processing infinite streams in Flink

Reading Time: 3 minutes Apache Flink is an open-source, distributed, Big Data framework for stream and batch data processing. Flink is based on the streaming first principle which means it is a real streaming processing engine and implements batching as a special case. Flink is considered to have a heart and it is the “Windows” operator. It makes Flink capable of processing infinite streams quickly and efficiently. Windows split Continue Reading

Spark: Streaming Datasets

August 18, 2020August 18, 2020Apache Spark, Big Data and Fast Data, Spark, Streaming, Studio-ScalaApache Spark, Big Data Analytics, DataFrame, DataStream API, Spark Streaming, Spark Structured Streaming, Streaming Spark

Reading Time: 3 minutes Spark providing us a high-level API – Dataset, which makes it easy to get type safety and securely perform manipulation in a distributed and a local environment without code changes. Also, spark structured streaming, a high-level API for stream processing allows us to stream a particular Dataset which is nothing but a type-safe structured streams. In this blog, we will see how we can create Continue Reading

Delta Lake: Schema Enforcement & Evolution

May 31, 2020June 1, 2020Apache Spark, Big Data and Fast Data, Java, SparkApache Spark, Big Data, Big Data Analytics, delta lake, schema

Reading Time: 4 minutes Nowadays data is constantly evolving and changing. As well as the business problems and requirements are evolving, the shape or the structure of the data is also changing. When that happens, we want to be in control of how the data or schema changes. But how we can achieve this? Delta Lake has good ways to control how schema changes. With Delta Lake, users have Continue Reading

Spark: createDataFrame() vs toDF()

May 23, 2020May 9, 2022Apache Spark, Big Data and Fast Data, Spark, Studio-ScalaApache Spark, Big Data Analytics, DataFrame, implicit methods, Methods, Spark with Scala, Tuples

Reading Time: 2 minutes There are two different ways to create a Dataframe in Spark. First, using toDF() and second is using createDataFrame(). In this blog we will see how we can create Dataframe using these two methods and what’s the exact difference between them. toDF() toDF() method provides a very concise way to create a Dataframe. This method can be applied to a sequence of objects. To access Continue Reading

Cluster vs Client: Execution modes for a Spark application

May 19, 2020August 30, 2022Apache Spark, Big Data and Fast Data, Java, Studio-ScalaBig Data Analytics, resourece manager, spark ecosystem, Spark Job, yarn

Reading Time: 3 minutes Whenever we submit a Spark application to the cluster, the Driver or the Spark App Master should get started. And the Driver will be starting N number of workers. Spark driver will be managing spark context object to share the data and coordinates with the workers and cluster manager across the cluster. Cluster Manager can be Spark Standalone or Hadoop YARN or Mesos. Workers will Continue Reading

Apache Spark: Delta Lake as a Solution – Part II

May 5, 2020October 12, 2020Analytics, Apache Spark, Big Data and Fast Data, Database, ML, AI and Data Engineering, Spark, Studio-Scala, Tech BlogsBig Data, Big Data Analytics, data analysis, delta lake

Reading Time: 3 minutes Well, we have already covered the missing features in Apache Spark & also the causes of the issue in executing Delta Lake in Part1. However, today we will be talking about What Delta Lake is & how it provides the solution to all those problems discussed herein Delta Lake as a Solution: Part1.As we all know that Spark is just a processing engine, it doesn’t Continue Reading

Apache Spark: Delta Lake as a Solution – Part I

May 4, 2020Analytics, Apache Spark, Big Data and Fast Data, Database, ML, AI and Data Engineering, NoSql, SQL, Streaming, Studio-Scala, Tech BlogsBig Data, Big Data Analytics, data analysis, delta lake

Reading Time: 3 minutes Today, everyone is talking about Delta Lake. Why? Ever tried to find the answer to this question? Yes or No doesn’t matter, don’t worry here in Part1 we will be discussing the same & also will be targetting the following questions: What are the features missing from Apache Spark? What kind of issues it causes in executing Data Lake? Answering the above questions will definitely Continue Reading

Top 7 Data Analytics and Management Trends for 2020

March 27, 2020October 12, 2020Big Data and Fast Data, Business Insights, Studio-ScalaAI, Analytics, Big Data Analytics, data analytics trends, NLP

Reading Time: 5 minutes We live in an era of data as it lies at the heart of digital transformation. And datasets are no longer as simple as before. They have increased in volumes, velocity, complexity and above all, are coming from multiple sources. Top tech giants like Google, Netflix, Amazon, and others are crunching massive amounts of data on a daily basis to give you a personalized experience. Continue Reading

Apache Spark: Tricks to Increase Job Performance

February 19, 2020Analytics, Apache Spark, Big Data and Fast Data, Database, ML, AI and Data Engineering, NoSql, Spark, SQL, Streaming, Streaming Solutions, Studio-ScalaApache Spark, Big Data, Big Data Analytics, data analysis, data engineering, fast data analytics, Spark Job, Spark Performance, Spark SQL, Spark Streaming

Reading Time: 2 minutes Apache Spark is quickly adopting the Real-world and most of the companies like Uber are using it in their production. Spark is gaining its popularity in the market as it also provides you with the feature of developing Streaming Applications and doing Machine Learning, which helps companies get better results in their production along with proper analysis using Spark. Although companies are using Spark in Continue Reading

Time Travel: Data versioning in Delta Lake

February 2, 2020February 2, 2020Analytics, Apache Spark, Big Data and Fast Data, Java, Spark, Studio-ScalaApache Spark, Big Data, Big Data Analytics, BigData, data lake, Data Management, data science, delta lake, Spark, Time Travel

Reading Time: 3 minutes In today’s Big Data world, we process large amounts of data continuously and store the resulting data into data lake. This keeps changing the state of the data lake. But, sometimes we would like to access a historical version of our data. This requires versioning of data. Such kinds of data management simplifies our data pipeline by making it easy for professionals or organizations to Continue Reading

Apache Spark: Read Data from S3 Bucket

January 7, 2020March 12, 2020Amazon, Analytics, Apache Spark, Big Data and Fast Data, Cloud, Database, ML, AI and Data Engineering, Spark, SQL, Studio-Scala, Tech BlogsAmazon S3, AWS, Big Data, Big Data Analytics, Big Data Storage, data analysis, fast data analytics

Reading Time: < 1 minute Amazon S3 Accessing S3 Bucket through Spark Edit spark-default.conf file You need to add below 3 lines consists of your S3 access key, secret key & file system spark.hadoop.fs.s3a.access.key “s3keys” spark.hadoop.fs.s3a.secret.key “yourkey” spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem

Apache Spark: Repartitioning v/s Coalesce

December 30, 2019Apache Spark, Big Data and Fast Data, Database, HDFS, ML, AI and Data Engineering, NoSql, Spark, Studio-ScalaAnalytics, Apache Spark, Big Data Analytics, data analysis, fast data analytics, partitioning in apache spark, real time analytics, Spark SQL

Reading Time: 3 minutes Does partitioning help you increase/decrease the Job Performance? Spark splits data into partitions and computation is done in parallel for each partition. It is very important to understand how data is partitioned and when you need to manually modify the partitioning to run spark applications efficiently. Now, diving into our main topic i.e Repartitioning v/s Coalesce What is Coalesce? The coalesce method reduces the number Continue Reading

Big Data Landscape explained

December 27, 2019December 27, 2019Apache Flink, Apache Kafka, Apache Spark, Big Data and Fast Data, Cassandra, Database, Elasticsearch, Flink, HDFS, Studio-ScalaBig Data Analytics, Big Data Storage, Big Data Tools, BigData

Reading Time: 5 minutes Big Data has now evolved into a buzz word and it seems everyone is either working on it or want to work on it. However, most of the people associate Big Data with some of the popular tool sets like Hadoop, Spark, NoSql databases like Hive, Cassandra , HBase etc. HDFS made Big Data popular as it gave us an option to distribute the data Continue Reading