Reading Time: 6 minutes There are various resources that give advice on how to [and how not to] partition your data, how to calculate the ideal file size, how to handle evolving schemas, how to build compaction routines, how to recover from failed ETL jobs, how to stream raw data into the data lake, etc. We have been working with customers throughout this time to encapsulate all of the Continue Reading
Reading Time: 3 minutes In today’s Big Data world, we process large amounts of data continuously and store the resulting data into data lake. This keeps changing the state of the data lake. But, sometimes we would like to access a historical version of our data. This requires versioning of data. Such kinds of data management simplifies our data pipeline by making it easy for professionals or organizations to Continue Reading
Reading Time: 3 minutes Data Lake – How to build a data lake and what are the phases involved in the same.
Reading Time: 3 minutes A data lake is a large storage repository that holds a vast amount of raw data in its native format until it is needed. Usually, the data in a lake consists of structured, unstructured and object data like pictures, blogs, posts, videos etc. An “enterprise data lake” (EDL) is simply a data lake for enterprise-wide information storage and sharing. Major stages of a data lake Continue Reading
