Microservice approach of sensor data integration


Formal Metadata

Microservice approach of sensor data integration
Title of Series
Part Number
Number of Parts
Bock, Nikolai (Hochschule Mainz - University of Applied Sciences)
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
FOSS4G, Open Source Geospatial Foundation (OSGeo)
Release Date

Content Metadata

Subject Area
The flexible integration of sensor data from several sources with different concepts is a interesting current research topic. This talk will show a solution which is developed as part of the “ActOnAir” project. The aim of “ActOnAir” is the development of a mobile recommendation system based on personalized environment sensor information in the e-health section. The talk focuses the sensor integration part. The system receives information from the following sources: mobile sensor devices (for particulate matter, ozone, etc.); activity trackers; human sensor information from smartphone apps. Additionally different online services for air quality data or weather information will be merged. Social Media information from concepts like “Collective Sensing” are also possible. Considering the interoperability of the sensor information the persistence of the information for data mining and visualization will be based on Sensor Web Enablement (SWE) standards. In respect to the flexibility and scalability, the presented approach will be based on microservices which provide the different sensor sources, processing steps for integration as well as sinks (such as a Sensor Observation Service (SOS)). The integration process can be defined in data streams using the provided services as modules. Also tapping into existing streams is possible. The deployment on different platforms will be easily supported, because of the microservice approach. During the talk the overall integration system concept, as well as the development of central modules, will be shown.
Mobile app Trail Thermodynamic system Real number Disintegration Division (mathematics) Motion capture Element (mathematics) Web 2.0 Factor analysis Measurement Computer animation Lie group Stress (mechanics) Integrated development environment Internet der Dinge
Mobile app Source code Standard deviation Disintegration Projective plane Source code Point cloud Bit Cartesian coordinate system Web 2.0 Web service Computer animation Web service Oval Central processing unit Subtraction Task (computing) Data integrity Computing platform Directed graph Directed graph Task (computing) World Wide Web Consortium
Trail Mobile app User interface Divisor Multiplication sign Mobile Web Disintegration Design by contract Mereology Thermodynamic system Factor analysis Measurement Latent heat Video game Stress (mechanics) Energy level Integrated development environment Gamma function Subtraction Mobile Web Mobile app Thermodynamic system Information Real number Element (mathematics) Projective plane Motion capture Cartesian coordinate system Element (mathematics) Data mining Computer animation Stress (mechanics) 5 (number) Coefficient Data type Data integrity
Mobile app Disintegration Volume (thermodynamics) Weight Workstation Web service Office suite Subtraction Computing platform Mobile Web Mobile app Thermodynamic system Information Cellular automaton Fitness function Workstation Cartesian coordinate system Flow separation Digital signal processing Computer animation Data acquisition Data storage device Web service Computing platform Cuboid Data integrity
Standard deviation State observer Keyboard shortcut Enterprise architecture Real number ACID Mereology Web 2.0 Web service Component-based software engineering Telecommunication Military operation Operator (mathematics) Queue (abstract data type) Information Communications protocol Identity management World Wide Web Consortium Enterprise architecture Standard deviation Multiplication Theory of relativity Information Military base Model theory Keyboard shortcut Representational state transfer Cartesian coordinate system Functional (mathematics) Component-based software engineering Message passing Computer animation Function (mathematics) Uniform resource name Web service Telecommunication Self-organization Communications protocol Reading (process)
Keyboard shortcut Batch processing Server (computing) Computer file Run time (program lifecycle phase) Disintegration Point cloud Mereology Thermodynamic system Variable (mathematics) Revision control Spring (hydrology) Web service Queue (abstract data type) Software testing Configuration space Message passing Run time (program lifecycle phase) Default (computer science) Enterprise architecture Thermodynamic system Point cloud Streaming media Variable (mathematics) Cartesian coordinate system Category of being Message passing Spring (hydrology) Computer animation Repository (publishing) Web service Computing platform Configuration space Data conversion Task (computing) Data integrity Asynchronous Transfer Mode Booting
Software engineering Digital filter Scheduling (computing) Computer file Multiplication sign Source code Adaptive behavior Point cloud Streaming media Function (mathematics) Thomas Kuhn Twitter Summation Web service Queue (abstract data type) Message passing Conditional-access module Subtraction Directed graph World Wide Web Consortium Newton's law of universal gravitation Source code Metropolitan area network Information Image registration Coprocessor File Transfer Protocol Twitter File Transfer Protocol Message passing Spring (hydrology) Computer animation Personal digital assistant Uniform resource name Function (mathematics) Data compression output Queue (abstract data type) Data type Directed graph Extension (kinesiology)
Point (geometry) Polygon Digital filter Transformation (genetics) Calculation Web browser Client (computing) Mereology Power (physics) Web service Term (mathematics) Object (grammar) Integrated development environment Scripting language Regular expression output Message passing Identity management Social class Raw image format Process (computing) Theory of relativity Information Scripting language Point (geometry) Polygon Basis (linear algebra) Client (computing) Letterpress printing Coprocessor Functional (mathematics) Symbol table Message passing Content (media) Computer animation Function (mathematics) Web service Uniform resource name output Object (grammar) Central processing unit Resultant Computer worm
Point (geometry) State observer Dataflow Asynchronous Transfer Mode Batch processing Injektivität Transformation (genetics) Run time (program lifecycle phase) Disintegration Insertion loss Function (mathematics) Summation Web service Hierarchy Operator (mathematics) Data mining Process (computing) Configuration space output Message passing Directed graph Task (computing) Social class Raw image format System dynamics Information Uniqueness quantification Coprocessor Variable (mathematics) Grass (card game) Batch processing Data mining Fluid statics Computer animation Web service Uniform resource name output Condition number Personal area network Supersonic speed Quicksort Task (computing) Data integrity
Integral transform Web service Computer animation Web service String (computer science) Directed graph Directed graph Maxima and minima
Dataflow Slide rule Run time (program lifecycle phase) Disintegration Computer-generated imagery Streaming media Commercial Orbital Transportation Services Medical imaging Spring (hydrology) Web service Bit rate Booting Task (computing) Graph (mathematics) Run time (program lifecycle phase) Mobile app Scaling (geometry) Dataflow Server (computing) Sound effect Streaming media Point cloud Image registration Process (computing) Computer animation Case modding Task (computing) Identity management Booting
State observer Transformation (genetics) Logarithm Multiplication sign Mobile Web Tap (transformer) Computer-aided design Temporal logic Mereology Emulation Web service Forest Computer worm Graph (mathematics) Default (computer science) Raw image format Information Scripting language Polygon Content (media) Streaming media Division (mathematics) Median Computer animation Uniform resource name Graph (mathematics) Data integrity
Dataflow Source code Parsing Mereology Weight Coprocessor Open set Spring (hydrology) Web service Data fusion output Task (computing) Source code Multiplication Spacetime Dataflow Information Projective plane Fitness function Point cloud Streaming media Representational state transfer Proof theory Spring (hydrology) Computer animation output Data fusion Task (computing) Booting Extension (kinesiology)
Hoax Control flow Perspective (visual) Semantics (computer science) Event horizon Web 2.0 Web service Meeting/Interview Bridging (networking) Software testing Software framework Graph (mathematics) Form (programming) System dynamics Process (computing) Information Structural load Projective plane Moment (mathematics) Bit Variable (mathematics) Cartesian coordinate system Connected space Category of being Message passing Computer animation Personal digital assistant Configuration space output Data type Directed graph
the so welcome once
again in this system In this track of dealing with
such a sensor web enablement or you may know Internet of Things and um 3rd so speaker is Nikolai book from offshore landmines and yeah like Simon
you will be talking about the practical application not just of the source standard but also I think of collection of 2 different data from different sensors and how to deal with that and while legal book will lead us through this yeah I know my name is Nikolai from the University of Applied Sciences in mind so and I will talk a bit about my customers approach of integrating the the data focusing on Cloud based migrants services and the the egg
agenda of of the talk will be given a short introduction to act and have a project the research is based on and I'm going to the integration platform for talking about the background little bit of micro-services on the Web enablement and then talking about these services sources but processes things and task at and the and of the orchestration of all these services is to combine them together and to latter piece the short and conclusion and future steps that
the act and it is all about knowledge handles may be more precise about asthma there are multiple factors that impact off off there at pollution and whether and individual factors like the chronicled the chronic diseases allergies and offer smoking and situational elements like your mom more momentarily constitution or stress level and to get all those together for a personal track it is not so easy and because we needed it for personal guidance we have to know where the person to which time was to get a life track of the user's constitution so
the EU project will divide in 4 main parts it's providing mobile mobile sensor for application to to get with you have cable and integration system for different than the types of contracts and the data mining and forecasting solution to to get the information on and specific user constitution and so at the and providing mobile app to get recommendations to yeah
so kind to the then the integration we have the data acquisition there are several concepts like and mobile sensor and then indoor and the hot door with so different animals like ozone and pollution and I'm like kid particulate matter and so on the temperature humidity down the mobile app which hasn't diary where you give the information of fewer domains like that you have an asthma attack or have you other and specific information done to this end application you can also appear devices like fitness instructors other things and then we have also the external services like personal data through the API should be considered like if you have you give a token to ask for your net capital API or Fitbit and we collect this data to the than data from in situ stations like that pollution from the federal environment office for example and and weather or climate information from from the services like storage of editing cells come from and so maybe could be interesting to put on collective sensing also inside so analyzing tradeoff or other services for example to get information out of this yeah
come to these and the data integration platform
1st will use short into introduction short information about what is the Microsoft architecture so when you thinking in models applications Manager can say OK I split my application into smaller decouples components regarding the functionalities of which which you can get a separated lived to each other so much so this is a separated in the sense of doubt about bases this is not used by another FIL you have easy reading real deployments because you have a dedicated functionality and you can also switch this functionality with the easy deployment and the communication is the 1 that provided to us through the REST API so that when you have the synchronous so where oral you use message queues when you have an acid news synchronous the data cake relations and then Madison is not only a technical part it's only it's also to see the the organization of of teams so you can say some of like a service should be responsive to them by a 2 pizza genes so it's a team which can be fitted by 2 pieces so it's like mostly 8 to 10 people looking I'm working this but so next part
is the than the Web enablement forward of the picture was shown on from in the 1st talk about this and that's a collection of OGC standards and the most part I'm using is sense of repr observation service with the consumer and producer operations we have the geospatial food operations and using the sensor data for persistent with binding of multiple protocols if
get to this the 1st the
service will be based on the a spring proved and sprinkle out to bring brute and provides to build up a little services so that was runtimes in the quick mode and you have the convention over configuration so you have default that the configurations or defaults which will be the find when you configure nothing more you can change this over configuration later on the configuration can be handled to properties and in property file or general files or through the Global Environment at the variables later with Frank larger get down the the binding to some micro to some the message queues like Kafka are and you register of the posts on that you get 10 test support for batch processing for example and all this is integrated with integration of messaging handling in the enterprise applications integration part so if you have only also cloud conflict which allows you to configure this is part here in the and the properties 5 has not to each service itself you can you can get a cloud config servers with a which is related to were based on gets hot enough for example and then you start yoga confirmation in Digital repository arbitrary and that it will be later on will be managed to configure this the we have 4
types of services for the 1st time will be the source of the which are type of inbound adaptive switch generate the messages in In most cases this will be posted by a polar and of the methods that will be sent to not put channel In our cases will be extended it's stupid creation piece where you consent century HTTP API your data you can send it to ask the user compressed so we will decompress it on the on the channels and also there are the API sources planned for that at pitch API and so from spring and this stream cloud flat-screen that also existing sources like and JDBC adapter or trigger which can trigger and the difference in schedules way you can input files or FTP server hall you can grapple with data from twitter for example that if the
extra 1 example for 4 of this source will be API source general reset we can maybe see that here's the main able binding with this we we will take 3 years message queue into place you can it's the source of info you define it as the source of which means you have child and in this case only the output channel where you can send the message later on In the
2nd term services will be the processes some processes will make manage message manipulation like transformers or feels those which will fit out some messages out of fear of some information but also you can get client request for example to requesting the UPS service or something like that that the end of the process of always have in part of the input and output channel so you get the information from the input channel and send later on to the other approaches there are several methods from former we will develop what we developed and for example that services this thing HTTP client to but the information from the in over the over and environmental data from the Federal Environmental Office I also want some developed and to transformer which will be the transform the point information to polygon From the isochrone putting cake relation to get reachability you huh the which part of of the city maybe I get in 5 minutes by power and this will be the result of a polygon and can be used to filter some can symbols later but there are also existing processes like transform of that should handle with scripts like Python JavaScript's Ruby or groovy there also expression-based filtering so on the basis of days ago or HDP Klein
give the example here is from the prone process for and you say in this part that you have the professor classes within the binding so you have an input channel and a lot of challenges that you define here at the transformer so you say this function get as variable and the payload of message from the input channel and the result is an object and the object will be the pot payload writing to by sending to the user and competent at the end this
this sort of will be the same soul of storing the data at the end in points they only get the data from the output from the unique put channel and uses outbound adapted so to to handle this summer 1 example will be you view them the observation services using the transactional from operations to stand the information to to the service from 2nd 1 will be our data mining which is based on the SNP hierarchy that and and there are also existing things like JDBC lock f 3 or 5 HDFS and so on yeah the
example of of the than the of observations service and nowhere furor in in 1 forcing and defined and integration flow so I will say OK and the grass disease information from that the input channels and send it to have another child is in them the the direct channel inside and then I can define here such flows to to say OK you crap from this journal to a transformation of transformer to generate in this kind of an observation class and filtered all those things which cannot be transformed so only get through the observations and at the end handle this by an insert observation handler which will be executed the inferred observations on request but the last 1 will be the task short all of you the task is is a service which we you have an end all the services that run until you stop and so on and the task will be started and the we ended up automatically you can use it for batch processing when you say OK every nite I want to grab hold information from the event variable from the then observation services and put it into HDFS or or put it into the SIP high enough for example yeah so you can
also composed services manually so you can say OK I started so this source of service of bounded into some channels and so on from a start up a professor service bonded to the things that journal doing this for the single as well and they're doing this on the
AWS or using Cuban leaders or something like that but this is complicated so How would would be to do with with the defined only if the strings center to an API and all the other things will die so for this
sprinkler dataflow comes into place on the full have several also build up on bring boot and this will manage all the stream definitions you make you rate registered your services has made an edge effects or talker images and then you define a stream but in a Unix-like always see it the next slide and it's you also can say OK I have a task and scale uses the the task runtime you can also stepped into existing
streams was used on the next slide also and once you have multiple did through deployed SPI is for example CloudFoundry job the pitcher John Cuba natives that or something you will be the yourself come to the
definition in here are some examples of the 1st division you say OK grab the time and doing some scripted scriptable transformation out of this in the 1st part of using Ruby and the 2nd using JavaScript or maybe huge using groovy and the default
and or data integration and this could be look like this you say OK i of the Act on http as the API so when doing some transformation on this and putting this into it as a form of that the effort of the world you see assumed can handle those and at the end of pushing the data into this and the observations so and also wrapped into this after this transformer wrapped the stream tap into it and doing some it in temporal jogging so you have only 1 the the content or 1 room for for 1 entry for afforestation and temporal information putting this into the graph of our has grown in story service to get a polygon uses polygon forest grabbing the environmental data from this and supporting this median and influx In the goal in last
month's light at the conclusion here we have the 1st proof of concept for processing different sources which is that it is and we using spring Buja and clustering for building up the you might services which is quite easy and the oxidation is through the spring data flow and we can control of this workflows using a Unix-like south through REST API the future work will be looking for all the space input sources like net at fit and so sources part 1 will be developed and that's CS part of the processor to to grab multiple climate services data than the data fusion fusing task will be extended to to handle more information like quality or or something like that and and then I want to intervene to investigate on the OGC seasonal things API to look at what can be done with it yet that was the
answer and and all open questions that you think is the amount I'm having already have here the 1st question from the great precision neglect but are you familiar with the open air quality project it's a it's a service that is being set up the Amazon Web Services and I just tweeted about from your because the danger that they could probably use the result of like the there are
only about an air quality of schema was used for example in the and 52 off as well about what they're going to build essentially a global crowdsourced air quality service but they need more In but so we we have the favorable we need input what are we going for a seminar and technical approach and maybe and later on we will get stated to but to get the data many people would the data they got the connection the Amazon city that placed here OK thank you more room for questions don't be afraid and well yes before you put in the detention SAS the sensor data how they handle the semantics then there are some of the type of sensor that you have that they need to do something in the SIS the new we tried to do with this so what you see as a sink so that some some configurations inside so you can do it in a static load and dynamic kids and say you define as enzyme L and send them to make an incident and Centaurus bridges sedans or request hence start of his service the land and putting all of the data on this sentence and the then on or you can say OK I want this and that dynamic most people you can grab the information out of the message and builders and that canal all of this and then reduced the sentence or on every message will be put to OK more question questions on Australia was it seems as you have a very generic framework for any kind of was a kind of processing not not even geo spatial processing so are you thinking of other types of applications 1 region because it was not just source yeah from from from the project's perspective uh with we tried it from the technical partners say OK it should be generic as generic as possible and we will tacit was test for example to show and use case but yet the problem from from technical perspective but it should be after that it is possible to say OK it's not how stated they found from from from the project's perspective you have you have to you have to have their cake event never environmental information to this event we will collect and then you've got a profile of of of your eye and so on and we can you can you can make a recommendation based of this for example but this can be In any event as can be you theoretically and data was around you so it's really describe it in an impressive summaries of 1 more question has arisen so then I am so thank you the variables of the census again and so you have a sense of books so running great bent or the experience they are not always and we know how our properties in developing of the sentiment so clean and not not at the moment have the status so it's like a bit in a bit of technically playing with fakes and then we will see if it works well with the real data coming from In the carried them we're almost at the break them the thank you again Nikolai and the other speakers and I forgot interested in Sensor Web Enablement so if you weren't it's assignment stock next week in Muenster and that's 52 north there is a sensor where people enable and conference Monday to Wednesday and a think monitor your tutorials but this is to ask if physical mark both which is here the form of conference room and that's really it's thank you for your attendance the good conferences


  668 ms - page object


AV-Portal 3.9.1 (0da88e96ae8dbbf323d1005dc12c7aa41dfc5a31)