We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Building an open source data lake at scale in the cloud

Formal Metadata

Title
Building an open source data lake at scale in the cloud
Title of Series
Number of Parts
490
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
This presentation will give an overview of the various tools, software, patterns and approaches that Expedia Group uses to operate a number of large scale data lakes in the cloud and on premise. The data journey undertaken by the Expedia Group is probably similar to many others who have been operating in this space over the past two decades - scaling out from relational databases to on premise Hadoop clusters to a much wider ecosystem in the cloud. This talk will give an overview of that journey and then describe the various open source components that Expedia Group have used and built to create multi-petabyte data lakes. These include existing open source projects like Hive, Hadoop, Terraform, Docker, Kubernetes as well as open source tools that we built to overcome some of the unexpected challenges we faced. The first of these is Circus Train — a dataset replication tool that copies Hive tables between clusters and clouds. We will also discuss various other options for dataset replication and what unique features Circus Train has. The second tool is Waggle Dance — a federated Hive metadata service that enables querying of data stored across multiple Hive metastores. We will then look at Apiary - a means to simplify the deployment of the various components of an open source data lake at scale including the Hive metastore, Waggle Dance, S3 bucket access, metadata change notifications and much more. We focus on actual problems and solutions that have arisen in a huge, organically grown corporation, rather than idealised architectures.