We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Faster Spark SQL: Adaptive Query Execution in Spark v3

Formale Metadaten

Titel
Faster Spark SQL: Adaptive Query Execution in Spark v3
Serientitel
Anzahl der Teile
637
Autor
Lizenz
CC-Namensnennung 2.0 Belgien:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Over the years, there has been extensive efforts to improve Apache Spark SQL performance. This talk will introduce the new Adaptive Query Execution (AQE) framework and how it can automatically improve user query performance. AQE leverages query runtime statistics to dynamically guide Spark's execution as queries run along. The talk will go over the main features in AQE and provide examples on how it can improve on the previous static query plans. Finally, we'll present the significant improvements we have seen on the TPC-DS benchmark with AQE. Examples of the new runtime optimizations include selecting the right join type (broadcast-hash-join vs. sort-merge-join), dealing with data skew, and automatically selecting the number of shuffle (reducer) partitions.