fast data analytics Archives

Stateful stream processing with Apache Flink(part 1): An introduction

Reading Time: 4 minutes Apache Flink, a 4th generation Big Data processing framework provides robust stateful stream processing capabilities. So, in a few parts of the blogs, we will learn what is Stateful stream processing. And how we can use Flink to write a stateful streaming application. What is stateful stream processing? In general, stateful stream processing is an application design pattern for processing an unbounded stream of events. Continue Reading

Flink: Join two Data Streams

December 3, 2020December 3, 2020Apache Flink, Big Data and Fast Data, Flink, Java#apache flink, Big, Big Data Analytics, fast data analytics, Flink, Flink Streaming, joins, Streaming, streaming analytics

Reading Time: 3 minutes Apache Flink offers rich sources of API and operators which makes Flink application developers productive in terms of dealing with the multiple data streams. Flink provides many multi streams operations like Union, Join, and so on. In this blog, we will explore the Window Join operator in Flink with an example. It joins two data streams on a given key and a common window. Let say we have one stream which contains salary information of all Continue Reading

Flink: Union operator on Multiple Streams

September 15, 2020September 16, 2020Apache Flink, Big Data and Fast Data, Flink, Java, Streaming, Streaming Solutions#apache flink, Big, Big Data Analytics, fast data analytics, Flink, Flink Streaming, Streaming, streaming analytics

Reading Time: 3 minutes Apache Flink offers rich sources of API and operators which makes Flink application developers productive in terms of dealing with the multiple data streams. Flink provides many multi streams operations like Union, Join, and so on. In this blog, we will explore the Union operator in Flink that can combine two or more data streams together. We know in real-time we can have multiple data streams from different sources Continue Reading

Re-evaluating Data Strategies to Respond in Real-Time

May 19, 2020October 12, 2020Big Data and Fast Data, Business Insights, ML, AI and Data Engineeringfast data, fast data analytics

Reading Time: 2 minutes Fast Data is empowering organizations to respond in real-time. About 75% of organizations are already using it for at least some of their applications.

Apache Spark: Tricks to Increase Job Performance

February 19, 2020Analytics, Apache Spark, Big Data and Fast Data, Database, ML, AI and Data Engineering, NoSql, Spark, SQL, Streaming, Streaming Solutions, Studio-ScalaApache Spark, Big Data, Big Data Analytics, data analysis, data engineering, fast data analytics, Spark Job, Spark Performance, Spark SQL, Spark Streaming

Reading Time: 2 minutes Apache Spark is quickly adopting the Real-world and most of the companies like Uber are using it in their production. Spark is gaining its popularity in the market as it also provides you with the feature of developing Streaming Applications and doing Machine Learning, which helps companies get better results in their production along with proper analysis using Spark. Although companies are using Spark in Continue Reading

Apache Spark: Read Data from S3 Bucket

January 7, 2020March 12, 2020Amazon, Analytics, Apache Spark, Big Data and Fast Data, Cloud, Database, ML, AI and Data Engineering, Spark, SQL, Studio-Scala, Tech BlogsAmazon S3, AWS, Big Data, Big Data Analytics, Big Data Storage, data analysis, fast data analytics

Reading Time: < 1 minute Amazon S3 Accessing S3 Bucket through Spark Edit spark-default.conf file You need to add below 3 lines consists of your S3 access key, secret key & file system spark.hadoop.fs.s3a.access.key “s3keys” spark.hadoop.fs.s3a.secret.key “yourkey” spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem

Apache Spark: Repartitioning v/s Coalesce

December 30, 2019Apache Spark, Big Data and Fast Data, Database, HDFS, ML, AI and Data Engineering, NoSql, Spark, Studio-ScalaAnalytics, Apache Spark, Big Data Analytics, data analysis, fast data analytics, partitioning in apache spark, real time analytics, Spark SQL

Reading Time: 3 minutes Does partitioning help you increase/decrease the Job Performance? Spark splits data into partitions and computation is done in parallel for each partition. It is very important to understand how data is partitioned and when you need to manually modify the partitioning to run spark applications efficiently. Now, diving into our main topic i.e Repartitioning v/s Coalesce What is Coalesce? The coalesce method reduces the number Continue Reading

Fast Data: The New Age Analytics For Enhanced Customer Experience

October 21, 2019July 24, 2020Big Data and Fast Data, Business Insightsfast data, fast data analytics, real time streaming data, streaming analytics

Reading Time: 6 minutes Data is evolving both in terms of quality and quantity in today’s enterprises and in the past few years, changes have occurred at a much faster pace. Not long ago, Big Data was considered the next big thing for digital transformation. Technologies like Hadoop and HBase made sense as batch processing of data was the norm. But things are not the same now. By the Continue Reading