Services

Go to Overview

 High performance systems Data strategy & Analytics Intelligence Driven Decisioning Platform strategy Blockchain Expert Services Academy, Audit & Consulting

Accelerators

Go to Overview

Platforms

KDP KDSP

Products

PremonR CodeSquad Studio9

TechHub

Akka Scala Rust Spark Functional Java Kafka Flink ML/AI DevOps Data Warehouse

Industries

Go to Overview

Travel

Retail Finance Healthcare Media and Publishing Consumer Internet Hi-tech & IOT

Insights

Go to Overview

 Case Studies Blogs Books Conferences & Events Resources OS contributions Webinars Knolx
CONTACT US

delta lake

Spark SQL in Delta Lake 0.7.0

Apache Spark, Big Data and Fast Data, Java, SQL, , , , ,

Reading Time: 3 minutes Nowadays Delta lake is a buzz word in the Big Data world, especially among the spark developers because it relegates lots of issues found in the Big Data domain. Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It is evolving day by day and adds cool features in its every release. Continue Reading

Delta Lake: Schema Enforcement & Evolution

Apache Spark, Big Data and Fast Data, Java, Spark, , , ,

Reading Time: 4 minutes Nowadays data is constantly evolving and changing. As well as the business problems and requirements are evolving, the shape or the structure of the data is also changing. When that happens, we want to be in control of how the data or schema changes. But how we can achieve this? Delta Lake has good ways to control how schema changes. With Delta Lake, users have Continue Reading

Apache Spark: Delta Lake as a Solution – Part II

Analytics, Apache Spark, Big Data and Fast Data, Database, ML, AI and Data Engineering, Scala, Spark, Tech Blogs, , ,

Reading Time: 3 minutes Well, we have already covered the missing features in Apache Spark & also the causes of the issue in executing Delta Lake in Part1. However, today we will be talking about What Delta Lake is & how it provides the solution to all those problems discussed herein Delta Lake as a Solution: Part1.As we all know that Spark is just a processing engine, it doesn’t Continue Reading

Apache Spark: Delta Lake as a Solution – Part I

Analytics, Apache Spark, Big Data and Fast Data, Database, ML, AI and Data Engineering, NoSql, Scala, SQL, Streaming, Tech Blogs, , ,

Reading Time: 3 minutes Today, everyone is talking about Delta Lake. Why? Ever tried to find the answer to this question? Yes or No doesn’t matter, don’t worry here in Part1 we will be discussing the same & also will be targetting the following questions: What are the features missing from Apache Spark? What kind of issues it causes in executing Data Lake? Answering the above questions will definitely Continue Reading

Spark: ACID Transaction with Delta Lake

Apache Spark, Big Data and Fast Data, Java, NoSql, Scala, Spark, , , , , ,

Reading Time: 3 minutes Spark doesn’t provide some of the most essential features of a reliable data processing system such as Atomic APIs and ACID transactions as discussed in the blog Spark: ACID compliant or not. Spark welcomes a solution to the problem by working with Delta Lake. Delta Lake plays an intermediary service between Apache Spark and the storage system. Instead of directly interacting with the storage layer, Continue Reading

Time Travel: Data versioning in Delta Lake

Analytics, Apache Spark, Big Data and Fast Data, Java, Scala, Spark, , , , , , , , ,

Reading Time: 3 minutes In today’s Big Data world, we process large amounts of data continuously and store the resulting data into data lake. This keeps changing the state of the data lake. But, sometimes we would like to access a historical version of our data. This requires versioning of data. Such kinds of data management simplifies our data pipeline by making it easy for professionals or organizations to Continue Reading

Diving deeper into Delta Lake

Apache Kafka, Apache Spark, Big Data and Fast Data, github, Scala, Spark, Streaming, Tech Blogs, , , , , ,

Reading Time: 6 minutes Delta Lake is an open-source storage layer that brings reliability to data lakes. It has numerous reliability features including ACID transactions, scalable metadata handling, and unified streaming and batch data processing.

Delta Lake To the Rescue

Apache Spark, Big Data and Fast Data, Java, python, Scala, Spark, Streaming, , , ,

Reading Time: 4 minutes Welcome Back. In our previous blogs, we tried to get some insights about Spark RDDs and also tried to explore some new things in Spark 2.4. You can go through those blogs here: RDDs – The backbone of Apache Spark Spark 2.4: Adding a little more Spark to your code In this blog, we will be discussing something called a Delta Lake. But first, let’s Continue Reading

OUR OFFERINGS

High Performace System Data Strategy & Analytics Intelligence Driven Decisioning Platform Strategy Blockchain Expert Services Academy, Audit & Consulting KDP KDSP PremonR

COMPANY

About Knoldus Knolway Case Studies Testimonials Careers Tech Expertise Conferences and Events News Room Our Partners CSR

LEARN

Why Knoldus? Blog Resources KnolX Webinars and Videos Podcast Rust Times Innovation Labs Knoldus Tech Hub Open Source Contributions

CONNECT

Contact Us Engagement Model Follow us Here:
Subscribe to our newsletter

Partners

lightbend
databricks
confluent
docker
hashicorp
ibm
© 2020 Knoldus Inc. All Rights Reserved.
Privacy Policy | Sitemap