We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Python: Winding Itself Around Datacubes

Formale Metadaten

Titel
Python: Winding Itself Around Datacubes
Untertitel
How to Access Massive Multi-Dimensional Arrays in a Pythonic Way
Serientitel
Anzahl der Teile
611
Autor
Lizenz
CC-Namensnennung 2.0 Belgien:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache
Produktionsjahr2017

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
While python has developed into the lingua franca in Data Science there isoften a paradigm break when accessing specialized tools. In particular for oneof the core data categories in science and engineering, massive multi-dimensional arrays, out-of-memory solutions typically employ their own,different models. We discuss this situation on the example of the scalable open-source arrayengine, rasdaman ("raster data manager") which offers access to and processingof Petascale multi-dimensional arrays through an SQL-style array querylanguage, rasql. Such queries are executed in the server on a storage engineutilizing adaptive array partitioning and based on a processing engineimplementing a "tile streaming" paradigm to allow processing of arraysmassively larger than server RAM. The rasdaman QL has acted as blueprint forforthcoming ISO Array SQL and the Open Geospatial Consortium (OGC) geoanalytics language, Web Coverage Processing Service, adopted in 2008. Notsurprisingly, rasdaman is OGC and INSPIRE Reference Implementation for their"Big Earth Data" standards suite. Recently, rasdaman has been augmented with a python interface which allows totransparently interact with the database (credits go to Siddharth Shukla'sMaster Thesis at Jacobs University). Programmers do not need to know therasdaman query language, as the operators are silently transformed, throughlazy evaluation, into queries. Arrays delivered are likewise automaticallytransformed into their python representation. The presenter is Principal Architect of rasdaman, editor of several "Big Data"standards, and co-chair of "Big Data" relevant working groups in several high-impact bodies. In the talk, the rasdaman concept will be illustrated with thehelp of large-scale real-life examples of operational satellite image andweather data services, and sample python code will be demonstrated live.