Big Data and Fast Data

Apache Beam: Side input Pattern

December 21, 2021August 30, 2022Apache Beam, Big Data and Fast Data, Cloud, ETL, JavaApache Beam, Big Data, Big Data Analytics, Big Data and Fast Data, data analysis, data engineering, pipeline

Reading Time: 3 minutes Apache Beam is a unified programming model for defining both batch and streaming data-parallel processing pipelines. It is a modern way of defining data processing pipelines. It has rich sources of APIs and mechanisms to solve complex use cases. In some use cases, while we define our data pipelines the requirement is, the pipeline should use some additional inputs. For example, In streaming analytics applications, it Continue Reading

Loading JSON data into Snowflake

December 15, 2020December 15, 2020AWS, Big Data and Fast Data, Cloud, Database, SnowflakeAI and Data Engineering, Analytics, AWS, Big Data and Fast Data, Cloud, Database, ml, snowflake, snowsql, sql

Reading Time: 4 minutes Have you ever faced any use case or scenario where you’ve to load JSON data into the Snowflake? We better know JSON data is one of the common data format to store and exchange information between systems. JSON is a relatively concise format. If we are implementing a database solution, it is very common that we will come across a system that provides data in Continue Reading