We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Warp10: A new paradigm for Time Series analysis

Formal Metadata

Title
Warp10: A new paradigm for Time Series analysis
Title of Series
Number of Parts
611
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language
Production Year2017

Content Metadata

Subject Area
Genre
Abstract
IoT changed deeply the value chain. End users of consumer devices aim to havean instant gratification which is based on data/metrics produced by theobject. At the end, IoT makers have to execute an epic split: Build the actualdevice (mechanical and electronics), Be over the top in firmware developmentin order to be secured Design services based on the object data. Surroundingall that is a hard job for a product team. In such a context Open SourceSoftware constitutes basic building blocks of IoT devices. Data produced bysmart devices are often time series, storing them is not the challenge, manygood open source solutions exists (OpenTSDB, Influx..) but developingalgorithms based on your data is difficult. It can become a nightmare if youhave to think about scalability. Warp10 is a 3 years old open source platformdesigned for collect, store and manipulate sensor data with WarpScript, alanguage dedicated to time series analysis. WarpScript works natively on timesseries stored into Warp 10 (based on either levelDB or HBase) but can beconnected to any data source. When you manipulate sensor data, you must dealwith privacy. Security and privacy have also been addressed by Warp 10 sinceits very inception, this includes fine grain access control mechanisms,encryption capabilities and throttling management. Warp10 plateform can beintegrated into an open ecosystem likes Storm, Flink, or also Apache Pig withone cornerstone, the capatibility of manipulate time series with WarpScript. The Warp 10 Platform is designed to collect, store and manipulate sensor data.Sensor data are ingested as sequences of measurements (also called timeseries). The Warp 10 Platform offers the possibility for each measurement toalso have spatial metadata specifying the geographic coordinates and/or theelevation of the sensor at the time of the reading. Those augmentedmeasurements form what we call Geo Time Series. The first differentiating factor of Warp 10 is that both space (location) andtime are considered first class citizens. Working with Geo Time Series allowsyou to have geo-located readings without having to use four separate seriesand having to keep track of the reading context. Complex searches like “find all the sensors active during last Monday in theperimeter delimited by this geo-fencing polygon” can be done without involvingexpensive joins between separate time series for the same source. We created WarpScript, an extensible stack oriented programming language whichoffers more than 700 functions and several high level frameworks to ease andspeed your data analysis. Simply create scripts containing your data analysiscode and submit them to the platform, they will execute close to where thedata resides and you will get the result of that analysis as a JSON objectthat you can integrate into your application. The WarpScript approach is another differentiating factor of Warp 10.Traditional time series platforms offer few manipulation options, usually onlyproviding a SQL like query language which cannot express complex analysis, orproviding a reduced set of aggregation functions. These approaches force youto produce more code on the client side thus increasing your development timeand leading to massive transfers of unprocessed data from the platform to yourapplications. Our approach lets you focus on your business use cases,simplifying IoT and sensor data applications by taking care of a larger chunkof the data analysis in a very efficient way.