We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Building a metadata ecosystem using the Hive Metastore

Formale Metadaten

Titel
Building a metadata ecosystem using the Hive Metastore
Serientitel
Anzahl der Teile
69
Autor
Lizenz
CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Metadata has been a key data infrastructure need since the beginning of our team's history at Stitch Fix. We began this journey in 2015 with the setup of the Hive Metastore to work with Spark, Presto, and the rest of the platform infrastructure. But as our business needs grew, we felt the need to enhance and extend our metadata ecosystem. In this talk, we want to share our journey of building additional capabilities with metadata to solve data and business challenges. Starting with our base infrastructure - the Hive Metastore, we will highlight each capability that led us to build the extensions into our present day metadata infrastructure. This includes improvements made to the Hive Metastore itself, extending the use of metadata beyond table schemas, and additional microservices we added to make access and use of metadata easier. Building these capabilities has helped our team use metadata to power internal use cases. We want to share how we went about building this ecosystem and the lessons we learned along the way.