Column-level lineage is coming to the rescue
Formal Metadata
| Title | Column-level lineage is coming to the rescue |
|
| Title of Series | |
| Number of Parts | 60 |
| Author | |
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. |
| Identifiers | |
| Publisher | |
| Release Date | |
| Language | |
Content Metadata
| Subject Area | |
| Genre | |
| Abstract | OpenLineage is a standard for metadata and lineage collection that is growing rapidly. Column-level lineage is one of its most anticipated features of the community that has been developed recently. In this talk, we:
* show foundations for column lineage within OpenLineage standard,
* provide real-life demo on how is it automatically extracted from Spark jobs,
* describe and demo column lineage extraction from SQL queries,
* show how the lineage can be consumed on Marquez backend.
We aim to provide demos to focus on practical aspects of the column-level lineage which are interesting to data practitioners all over the world. |
|