We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Migrate Data, Mesh in mind

Formal Metadata

Title
Migrate Data, Mesh in mind
Title of Series
Number of Parts
60
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date2023
LanguageEnglish

Content Metadata

Subject Area
Genre
Abstract
For quite some time, Hadoop served as the data warehouse for Kleinanzeigen. However, the central teams eventually decided to say goodbye to this old friend due to its outdated nature and high costs. This migration presented us with a valuable opportunity to embrace the Data Mesh strategy and establish a new data pipeline. In this presentation, our objective is to provide an overview of our approach, which involves implementing a cloud-based data pipeline with the help of dbt and Airflow. Furthermore, we will delve into the challenges we faced during the process, including the debugging of legacy data flows, the complexities of copying data to s3, and dealing with the domain ownership issues. By sharing these experiences, we aim to provide valuable insights into our journey.