We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Machine learning on non curated data

Formal Metadata

Title
Machine learning on non curated data
Subtitle
Dirty data made easy
Title of Series
Number of Parts
118
Author
License
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
According to industry surveys [1], the number one hassle of data scientists is cleaning the data to analyze it. Textbook statistical modeling is sufficient for noisy signals, but errors of a discrete nature break standard tools of machine learning. I will discuss how to easily run machine learning on data tables with two common dirty-data problems: missing values and non-normalized entries. On both problems, I will show how to run standard machine-learning tools such as scikit-learn in the presence of such errors. The talk will be didactic and will discuss simple software solutions. It will build on the latest improvements to scikit-learn for missing values and the DirtyCat package [2] for non normalized entries. I will also summarize theoretical analyses in recent machine learning publications. This talk targets data practitioners. Its goal are to help data scientists to be more efficient analysing data with such errors and understanding their impacts. With missing values, I will use simple arguments and examples to outline how to obtain asymptotically good predictions [3]. Two components are key: imputation and adding an indicator of missingness. I will explain theoretical guidelines for these, and I will show how to implement these ideas in practice, with scikit-learn as a learner, or as a preprocesser. For non-normalized categories, I will show that using their string representations to “vectorize” them, creating vectorial representations gives a simple but powerful solution that can be plugged in standard statistical analysis tools [4]. [1] Kaggle, the state of ML and data science 2017 https://www.kaggle.com/surveys/2017 [2] https://dirty-cat.github.io/stable/ [3] Josse Julie, Prost Nicolas, Scornet Erwan, and Varoquaux Gaël (2019). “On the consistency of supervised learning with missing values”. https://arxiv.org/abs/1902.06931 [4] Cerda Patricio, Varoquaux Gaël, and Kégl Balázs. ""Similarity encoding for learning with dirty categorical variables."" Machine Learning 107.8-10 (2018): 1477 https://arxiv.org/abs/1806.00979