Join optimization

Spark 3.0 – Adaptive Query Execution With Example

June 3, 2022June 6, 2022sparkJoin optimization, performance optimization, Spark

Reading Time: 4 minutes Introduction Adaptive Query Execution (AQE) is one of the greatest features of Spark 3.0 which reoptimizes and adjusts query plans based on runtime statistics collected during the execution of the query. Need of AQE With each major release of Spark, it’s been introducing new optimization features in order to better execute the query to achieve greater performance. Before spark 3.0, cost-based optimization uses table statistics to determine the Continue Reading

Apache Spark’s Join Algorithms

August 31, 2020August 31, 2020Apache Spark, Studio-Scala, Tech BlogsApache Spark, Broadcast Join, Join opertaions, Join optimization, Joins in Spark, Shuffled Hash Join, Sort Merge Join

Reading Time: 4 minutes Joins in Apache Spark are fundamental transformations, but if you are not familiar with their internal algorithm, they can become too expensive.