We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Data Formats for Data Science

Formale Metadaten

Titel
Data Formats for Data Science
Serientitel
Teil
84
Anzahl der Teile
169
Autor
Lizenz
CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
Valerio Maggio - Data Formats for Data Science The CSV is the most widely adopted data format. It used to store and share *not-so-big* scientific data. However, this format is not particularly suited in case data require any sort of internal hierarchical structure, or if data are too big. To this end, other data formats must be considered. In this talk, the different data formats will be presented and compared w.r.t. their usage for scientific computations along with corresponding Python libraries. ----- The *plain text* is one of the simplest yet most intuitive format in which data could be stored. It is easy to create, human and machine readable, *storage-friendly* (i.e. highly compressible), and quite fast to process. Textual data can also be easily *structured*; in fact to date the CSV (*Comma Separated Values*) is the most common data format among data scientists. However, this format is not properly suited in case data require any sort of internal hierarchical structure, or if data are too big to fit in a single disk. In these cases other formats must be considered, according to the shape of data, and the specific constraints imposed by the context. These formats may leverage *general purpose* solutions, e.g. [No]SQL databases, HDFS (Hadoop File System); or may be specifically designed for scientific data, e.g. hdf5, ROOT, NetCDF. In this talk, the strength and flaws of each solution will be discussed, focusing on their usage for scientific computations. The goal is to provide some practical guidelines for data scientists, derived from the the comparison of the different Pythonic solutions presented for the case study analysed. These will include `xarray`, `pyROOT` *vs* `rootpy`, `h5py` *vs* `PyTables`, `bcolz`, and `blaze`. Finally, few notes about the new trends for **columnar databases** (e.g. *MonetDB*) will be also presented, for very fast in-memory analytics.