Knoldus Blogs

Google Cloud Platform: Migrating Data to New Schemas on Big Query Using Dataflow

February 1, 2023February 14, 2023AI, GCP, ML, AI and Data EngineeringApache Beam, BigQuery, Data, Data Migration, Dataflow, Google Cloud, Pipelines, python

Reading Time: 6 minutes Migrating data on Google Cloud BigQuery may seem like a straightforward task, until you run into having to match old data to tables with different schemas and data types. There are many approaches you can take to moving data, perhaps using SQL commands to transform the data to be compatible with the new schema. However, SQL has limitations as a programming language, being a query-centric Continue Reading

An Overview of Apache Beam Features

June 30, 2022July 1, 2022Javaapache, Apache Beam, Functional Programming, java, Java 8

Reading Time: 3 minutes We’ll talk about Apache Beam in this guide and discuss its fundamental concepts. We will begin by showing the features and advantages of using Apache Beam, and then we will cover basic concepts and terminologies. Ever since the concept of big data got introduced to the programming world, a lot of different technologies and frameworks have emerged. The processing of data can be categorized into Continue Reading

Introduction to Apache Beam

June 23, 2022June 23, 2022Apache Beam, Big Data and Fast Data, Database, Tech BlogsApache Beam

Reading Time: 3 minutes What is Apache Beam? Apache Beam is a unified programming model for batch and streaming data processing jobs. It provides a software development kit to define and construct data processing pipelines as well as runners to execute them. Apache Beam is designed to give a portable programming layer. The Beam Pipeline Runners translate the data processing pipeline into the API compatible with the back-end of the user’s Continue Reading

Apache Beam Core Transforms

June 21, 2022June 21, 2022Apache BeamApache Beam

Reading Time: 6 minutes Introduction Transform in Apache Beam are the operations in your pipeline, and provide a generic processing framework. You provide processing logic in the form of a function object (colloquially referred to as “user code”), and your user code is applied to each element of an input PCollection (or more than one PCollection). Core Beam transforms Beam provides the following core transforms, each of which represents a different processing Continue Reading

Apache Beam: Ways to join PCollections

December 27, 2021August 30, 2022Apache Beam, Big Data and Fast Data, JavaApache Beam, Big Data Analytics, Join opertaions, joins

Reading Time: 4 minutes Joining multiple sets of data into a singular entity is very often when working with data pipelines. In this blog, We will cover how we can perform Join operations between datasets in Apache Beam. There are different ways to Join PCollections in Apache beam – Extension-based joins Group-by-key-based joins Join using side input Let’s understand the above different way’s to perform Join with examples. We Continue Reading

Apache Beam: Side input Pattern

December 21, 2021August 30, 2022Apache Beam, Big Data and Fast Data, Cloud, ETL, JavaApache Beam, Big Data, Big Data Analytics, Big Data and Fast Data, data analysis, data engineering, pipeline

Reading Time: 3 minutes Apache Beam is a unified programming model for defining both batch and streaming data-parallel processing pipelines. It is a modern way of defining data processing pipelines. It has rich sources of APIs and mechanisms to solve complex use cases. In some use cases, while we define our data pipelines the requirement is, the pipeline should use some additional inputs. For example, In streaming analytics applications, it Continue Reading

Stateful processing with Apache Beam

December 20, 2021December 20, 2021JavaApache Beam, Apache-Beam, Java8, stateful streaming

Reading Time: 6 minutes Overview Beam lets us process unbounded, out-of-order, global-scale data with portable high-level pipelines. Stateful processing is a new feature of the Beam model that expands the capabilities of Beam. With these new features, we can unlock newer use cases and newer efficiencies Quick Recap In Beam, a big data processing pipeline is a directed, acyclic graph of parallel operations called PTransforms processing data from PCollections. The boxes are PTransforms and the edges Continue Reading