We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Column-level lineage is coming to the rescue

Formal Metadata

Title
Column-level lineage is coming to the rescue
Title of Series
Number of Parts
60
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date2023
LanguageEnglish

Content Metadata

Subject Area
Genre
Abstract
OpenLineage is a standard for metadata and lineage collection that is growing rapidly. Column-level lineage is one of its most anticipated features of the community that has been developed recently. In this talk, we: * show foundations for column lineage within OpenLineage standard, * provide real-life demo on how is it automatically extracted from Spark jobs, * describe and demo column lineage extraction from SQL queries, * show how the lineage can be consumed on Marquez backend. We aim to provide demos to focus on practical aspects of the column-level lineage which are interesting to data practitioners all over the world.