Building a multi-purpose platform for bulk data using sqlalchemy
Formal Metadata
| Title | Building a multi-purpose platform for bulk data using sqlalchemy |
|
| Title of Series | |
| Part Number | 40 |
| Number of Parts | 173 |
| Author | |
| License | You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license. |
| Identifiers | |
| Publisher | |
| Release Date | |
| Language | |
| Production Place | Bilbao, Euskadi, Spain |
Content Metadata
| Subject Area | |
| Genre | |
| Abstract | Christian Trebing - Building a multi-purpose platform for bulk data using sqlalchemy
At Blue Yonder, we've built a platform that can accept and process
bulk amounts of data for multiple business domains (e.g. handling
retail store location and sales data) using SQLAlchemy as a database
abstraction layer.
We wanted to use as much of SQLAlchemy as possible, but we quickly
found that the ORM (Object Relational Mapper) is not suitable for
handling large amounts of data at once. At the same time, we did not
want each team of developers working on individual business domains to
have to handcraft their own SQL statements. To solve this problem, we
built an application configuration that closely resembles an
SQLAlchemy model, but also contains application-specific logic
settings.
In this talk I will demonstrate:
- an application architecture for multiple business domains
- the structure of the domain configuration utilized in the
generation of the SQLAlchemy model, SQLAlchemy core statements, and
other application functionality
- how the domain configuration is used throughout the application
(consuming and parsing incoming data, storing it in a database and
ensuring data quality) |
|
| Keywords | |