Description:We describe a system that transforms sequences of MODIS images covering the entire Earth into time-optimized data cubes to provide rapid access to time series data for various applications.Abstract:Satellite time series data are key to global change monitoring related to climate and land cover change. Various research and operational applications such as crop monitoring and fire history analysis rely on rapid access to extended, hyper-temporal time series data. However, converting large volumes of spatial data into time series and storing it efficiently is a challenging task. In order to solve this Big Data problem, CSIR has developed a system which is capable of automated downloading and processing of several terabytes of MODIS data into time-optimized "data cubes." This time series data is instantly accessible via a variety of applications, including a mobile app that analyzes and displays 14 years of vegetation activity and fire time series data for any location in the world. In this presentation we will describe the implementation of this system on a high-performance Storage Area Network (SAN) using open source software including GDAL and HDF5. We discuss how to optimally store time series data within HDF cubes, the hardware requirements of working with data at this scale as well as several challenges encountered. These include writing high-performance processing code, updating data cubes efficiently and working with HDF data in a multi-threaded environment. We conclude by showing visualizations of our vegetation and burned area time series data in QGIS, web apps, and mobile apps. |