istSOS: latest developments and first steps into the OSGeo incubation process
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 183 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/32135 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Place | Seoul, South Korea |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G Seoul 2015128 / 183
7
8
47
53
54
65
73
74
79
82
84
92
102
103
105
124
126
127
130
141
142
143
156
161
162
170
176
178
181
183
00:00
Process (computing)Open source19 (number)BitService (economics)Standard deviationQuicksortBitProjective planePhysical systemProcess (computing)ImplementationMultiplication signState observerSummierbarkeitWeb serviceSoftwareComputer animation
00:51
Computer networkData managementSoftwareMultiplication signPhysical systemComputer animation
01:17
MeasurementData transmissionPhysical systemAnalogyModemReal numberStandard deviationOpen setMaxima and minimaContext awarenessKeyboard shortcutRevision controlService (economics)Personal digital assistantInterface (computing)Operations researchIndependence (probability theory)Normed vector spaceType theoryMenu (computing)QuantumFormal languageInternet der DingeBlock (periodic table)Virtual machineServer (computing)InternetworkingExecution unitOpen sourceSoftwareLinear multistep methodSurjective functionSimultaneous localization and mappingMobile WebPoint (geometry)Landau theoryFunction (mathematics)File formatBitRight angleField (computer science)Different (Kate Ryan album)DialectState of matterClient (computing)Multiplication signPhysical systemWordUniform resource locatorWater vaporStaff (military)SoftwareVirtual machineWorkstation <Musikinstrument>CondensationLibrary (computing)Open setScheduling (computing)Type theoryObject (grammar)Computer architectureStandard deviationLevel (video gaming)ImplementationPosition operatorCore dumpPresentation of a groupPoint (geometry)View (database)Endliche ModelltheoriePredictabilityDependent and independent variablesMereologyInternet der DingeWeb serviceSpeech synthesisOpen sourceGraphical user interfaceMetadataTelecommunicationTime zoneInformationInterface (computing)Extension (kinesiology)Kritischer Punkt <Mathematik>Mobile WebIncidence algebraResultantMeasurementState observerComputer fileAreaMedical imagingCollaborationismMultiplicationSystem administratorDecision theoryCellular automatonSpacetimeFunction (mathematics)Cartesian coordinate systemProcedural programmingConfiguration spaceTransformation (genetics)Insertion lossFile formatInternetworkingRevision controlRepresentational state transferWeb 2.0VorwärtsfehlerkorrekturBuildingDenial-of-service attackTask (computing)Computer animation
09:40
Graphical user interfaceData managementInterface (computing)File formatFunction (mathematics)Personal identification numberIntrusion detection systemComputer wormMomentumAverageSummierbarkeitParameter (computer programming)Latent class modelPhysical lawMassLinear multistep methodUniform resource nameConstraint (mathematics)Category of beingLatent heatConfiguration spacePrice indexReal numberAsynchronous Transfer ModeMaxima and minimaoutputExecution unitTelephone number mappingSpecial unitary groupInsertion lossEvent horizonTime zoneNormed vector spaceVirtual realityThermal radiationScalable Coherent InterfaceWeb pageError messageSystem administratorScheduling (computing)Broadcast programmingServer (computing)Computer fileBenachrichtigungsdienstWeb 2.0AreaMultiplication signSound effectTime zoneComputer configurationThermal radiationWorkstation <Musikinstrument>Noise (electronics)Complex analysisInheritance (object-oriented programming)Performance appraisalTheoryBitSocial classNatural numberView (database)Web serviceMathematicsWordPoint (geometry)State observerAverageGradientMathematical analysisNichtlineares GleichungssystemTwitterImplementationField (computer science)1 (number)Condition numberStandard deviationCategory of beingTransportation theory (mathematics)Power (physics)Order (biology)Function (mathematics)Transformation (genetics)File formatHome pageRange (statistics)Latent heatServer (computing)Subject indexingDependent and independent variablesTranslation (relic)Functional (mathematics)Universe (mathematics)Open sourceTime seriesProjective planeSoftwareInformationMereologyProcedural programmingSummierbarkeitFreezingProcess (computing)Perspective (visual)Form (programming)Real-time operating systemSoftware testingCartesian coordinate systemVideo gameSystem administratorScheduling (computing)Error messageMeasurementDatabaseCurveTelecommunicationDegree (graph theory)Parameter (computer programming)EmailStatisticsComplete metric spaceOutlierEvent horizonArithmetic meanInterface (computing)FrequencyCalculationQuicksortCovering spaceInstance (computer science)Extension (kinesiology)Text editorTheory of relativityScripting languageComputer animation
18:03
SoftwareProjective planeException handlingMereologyProcess (computing)SoftwareMultiplication sign
18:35
Process (computing)WhiteboardVotingTelecommunicationCodeSpecial unitary groupDrum memoryContent (media)Computer networkDistribution (mathematics)Standard deviationWebsiteMaxima and minimaPersonal identification numberSeries (mathematics)Menu (computing)Complete metric spaceSoftware developerScalabilityExecution unitOffice suiteImplementationStandard deviationTelecommunicationFunctional (mathematics)CodebuchLine (geometry)Mathematical analysisRight angleSystem callSummierbarkeitLibrary (computing)Maxima and minimaInformationInterface (computing)Extension (kinesiology)Natural languageWebsiteData miningControl flowParameter (computer programming)Row (database)File formatProcess (computing)Speech synthesisScalabilityState observerMetropolitan area networkEndliche ModelltheorieDifferent (Kate Ryan album)Point (geometry)Projective planeComponent-based software engineeringForm (programming)VorwärtsfehlerkorrekturCore dumpSign (mathematics)Reading (process)Web serviceClient (computing)Software developerWeb 2.0Water vaporWhiteboardOpen sourceProcedural programmingCalculationTime seriesSharewareFront and back endsFunction (mathematics)SoftwareDecision theoryGeneric programmingComputer animation
24:30
BenachrichtigungsdienstThermal radiationVirtual realityMoment (mathematics)Link (knot theory)Figurate numberServer (computing)System callInterface (computing)Parameter (computer programming)Process (computing)Direction (geometry)Social classState observerProcedural programmingMereologyElectronic data processingRight angleNormal (geometry)VorwärtsfehlerkorrekturDependent and independent variablesError messageComputer animation
25:33
Virtual realityComputer animation
Transcript: English(auto-generated)
00:04
Hello, it's a little bit of time that we are around with this implementation of the sensor observation services. Some of you may already have seen this and it's a sort of review of the projects and some acknowledgement about the incubation
00:23
process that we started with in OSGU. So first of all we had workshops last day, it was really fun, we were really happy, at least me, because we have enough people, interested people, we play with the software, we play with some sensor
00:46
Arduino, upload the data and ingest the data in the systems. Like here is the pictures, just connected a temperature sensor to Arduino and the ESO software.
01:03
So everything started a long time ago because we have the need to maintain the monitoring system for the Canton-Ticino in southern Switzerland, hydrometeorological networks, and so in 2010 there was a switch from the
01:21
sensors that were communicated with SMS, basically, we introduced the GPR, actually, the administration space, to integrate with GPRS sensors and we started to think maybe we could use something more smart and more intelligent to get the data, archive and serve the data. So we started looking at the sensor
01:47
observation services and yeah, I have to say also that one of the main applications that we have is about lake flooding and this lake in the bottom
02:02
right is transboundary, so we have to mesh, fuse data from different networks, getting data from Italian partners to regions, getting data from network from the confederations, for the meteorological office, then our own
02:21
networks and put all this data together. So we thought that the standard could be a viable solution to share the data, to get the data and share the data more easily. Then we start to look for solutions and we wanted something that was straightforward, standard, open and possibly Python, because we were
02:45
started using Python and using this type of technology and we didn't found actually something that was satisfying for us in the market, so we start to develop our own implementations. This is basically the concept of the sensor
03:01
observation services, but I think the presentation before already explained what is it, from a point of view generally there are two type of user to this sensor, there is data consumer and data producers, the sensor and the people that use this data. The general requests I think are really well done
03:22
and the principle is to explore the service to get an understanding of what is available and then get metadata of the sensors and then start to grab the data of these sensors. On the other side, the sensor should be able to register through the networks and then start to send information and data to
03:41
the service. I put this one, could it be an enable for the Internet of Things, the sensor observation services, somehow is a standard, somehow facilitate the communications and the understanding and the usable of this data. We will see,
04:02
we have seen that there is a new proposal for a new standard that I found really interesting and yeah, I will discuss also more with reference to the previous talk. Coming to the software, we have implemented in GPL version 2 and
04:23
above license and we have developed it in strict collaboration with our colleagues. We work at the health science institutes and most of the time we work to support hydrologists, hydrogeologists, geologists. So this
04:45
monitoring network was in place already managed by our institute since 15 years. So people managing the network has a lot of experience, has the best practice, the best procedures, so we need to, we needed to find and develop a
05:04
solution that fits their needs. Up to now we support only two types of sensors and we define it somehow, this type of sensor, so sensor in the field in C2, not remote sensor, not something that is sensing remote objects and
05:26
with fixed or mobile position, location, something a meteorological station in the fields or maybe higher pollution sensor on a vehicle and point-wise, we don't have measurements distributed. This is how is the architecture,
05:47
basically we have a core part that implements the standard where we have data warehouse, Postgres, PostGIS, then we have some, an ESOS library that is the core
06:01
that enhances to the ODC standard in XML and is written in Python. We have then some configuration files and on top of this we build up a web administration library that expose ARS full services that communicate in JSON,
06:22
basically. So we can have serve three type of users, a SOS client, pure SOS client that talk with us in XML or a user that use an interface build that you make extensive use of
06:41
the web administration library to using to interact with the system or a machine user that use the REST API to perform a scheduled task. The web service is a OSGI
07:01
Python adapter and Apache. As I said, we need to respond to some needs and some special features that were requested to us and that was not included in the standard. So somehow we develop some special features, some extensions, let's say, to the standards and we better code
07:25
them special features and to make this solution more usable and more effective. We design the graphical interface so there is no need to handle with code or anything, just using the interface. Then we support multiple output formats like JSON but also CSV
07:47
because hydrologists use 90% of the time CSV. We introduce data aggregations because we thought that it should be wise at least to reduce the data before sending the data to the user
08:07
instead of sending much more data and then let the user reduce it. And we support time zone because you can be in the situation where data are shared between different nations,
08:22
different time zones, and networks happen somewhere but you want to know in your local time when it happens. And then we support, natively, a quality index. This is not specific, you can do but it's not specifically designed in the sensor observation
08:44
services but because our data were used for emergency and our data are used by models that make some predictions that are used in emergency, we want the modelers to know the quality of the
09:07
of our data and the certainty of the model result before taking any decision. I think this is one of the critical points we need to know the quality of our data because if our data is wrong
09:20
our decision may be wrong. Then we support also virtual procedure. Virtual procedure are some procedures that actually does not exist but are based on transformation on other procedures. So the motivation why we started is hydrologists. In a river they measure river hate
09:48
but then people use river discharge. There is an equation writing curve that transforms the hate of the river in discharge but this curve is often measured also after having
10:02
collecting information. So hydrologists go out with topography, measure the topography of the bed of the river, calculate the new equation, say okay use this equation for the last three months. So we store only hate data and then we convert on the fly but from the user perspective
10:21
the user just sees a sensor that is measured in discharge in SOS format. So I will go through these small features like the web administrations. There is a part where the user can
10:43
configure his services and I have to say I see Andrea and we took inspiration also from your service because we thought that this is one of the key points of making these technologies usable, having an interface that enables a lot of users to interact. And then we develop also
11:04
data editors where people can look at time series, can edit the values, can make calculators. For example in the tutorial we showed that we have a station with the missing data periods and we feel it making the average of the two nearest stations for instance. Output format
11:29
there is CSV, we defined how our own let's say CSV with a header containing the URI of the observer properties, then this is the SOS get observation request. But then there is
11:45
also the JSON response that is basically one-to-one translation of the XML standard. Aggregation on the fly, this is an example, we can ask data, we can add a couple of
12:03
specific functions and get the data aggregated. Another thing is I was talking about this quality check. In this source each measure that is inserted in the database gets a quality index.
12:21
This quality index can be inserted by the user but we implemented also two quality check procedures within the sensor observation services. So if you configure your interface, your service, your sensor, you can pass through these two quality check tests in real time
12:44
when you insert the data. So the first check is the evaluation of the measures. If you are measuring rainfall it cannot be negative for example, or if you're measuring temperature it cannot be 150 Celsius degree. The second one is a test within statistical values. So you
13:07
basically detect if it is not an outliers and this check is configured for a given observer property and a given sensor. So the first check is temperature should have a reasonable
13:24
values. The second one is temperature in CEUL should have generally disarranged values. And then there are other procedures that we apply on our database on our server but they are external because you need to have a complete time series or part of the time series to detect
13:42
if there is some peak in the time series so this is probably an error or you need to compare with nearby station and then you need data also from the others so these other steps are run outside. This is an example when you insert these values and you insert the quality index
14:03
you get this quality index. This quality index are already defined in the SOS resource applications but if you want you can override and define your own time zone support.
14:20
When you ask data in a get observation request and you use your time zone event time zone then the response is given in the time zone that you have requested. There is no extra parameter just specifying the time zone in the get observation request. This is the virtual
14:43
procedures. This is an example in a European project in precise farming where there are sensor meteorological extension in the fields. They were observing temperature, relative humidity, wind speed and solar radiation but actually then people wanted to do the reference evapotranspiration.
15:05
So we make a small script. We create a virtual procedure so the user see that there is a sensor that is sensing the reference evapotranspiration. This is calculated on the flame base on original values. So if you change the values you change also the evapotranspiration calculate.
15:29
This is a little bit experimental. We have implemented a status page because when you have a lot of sensor then you have to manage a network. It's difficult to see it, to understand
15:41
there is a sensor that is now sending data, how long, why etc. So this is a sort of dashboard to have an idea of which sensor is not really working. Then you can check and say okay this blue meter is not working because it's above 2000 meters and in winter there is no cover
16:02
and the administration can say it's okay or try to find out. We set up an acquisition job scheduler internally because there was a question about the sensor, why they do not support
16:21
actually the standard etc. There is a vertical market from the sensor vendor so they when they sell the sensor they send also their format and their application to get the data and this is what is really happening also in our network. So we use their technology to get the data
16:42
because there is no other options many times, not always but very often. Then we have some internal scheduler that gets the data and pushes the data into our sensor observation services.
17:00
We implemented a notification service that is not really the implementation of OGC proposal of a web notification service standard but it follows somehow the principle, let's say, then it is done on RESTful and JavaScript and
17:25
and JSON but basically one can set up some some triggering and in python he can he can develop his own complex functions or he can
17:40
develop something simple and we provide already the classes to get rise mean alert when these three sensors, the average of these three sensors temperature is higher than these values and these are up to now send out SMS and Twitter but it can be easily extended to other mode of communication. Then we start to enter the incubation part. We start to enter incubation
18:10
less than one year ago I think and the first part is keep calm. It's not a short process obviously probably depend on the status of your project or the software etc if
18:25
you are well but it also relies on volunteers works of the OCL so you can ask them to work every day on your project so it keeps time to start the project this is also to maybe to
18:41
explain how it works you have to submit the proposal then the committee review the recommendations and decide if you can enter or ask you for some modification then you have to find a mentor that is your tutor let's say then check make a notion to the board and the board
19:04
board and then if everything is okay then you are into incubation what are the requirements basically to be open what was your check in incubating projects the copyright and the licenses
19:20
if your project is correct and doesn't use makeup use of some licenses etc then you check the documentation if it is available and you release procedure of your software the processes that you use and check if your community is healthy and active
19:46
where we are we did the code but we start the code provenance to check every line of code if everything is fine i think that we are quite good also because most of the code we write on
20:00
our own and we don't rely too much on other libraries then we develop a website with some information we set up a demo and some download and etc we develop the documentation
20:23
from the user but also for the developers either for the libraries the base library also for the restful library then we start to set up our release
20:40
devian packages and then what's next finalize the code provenance completes developer docs there are some governance that has to be defined for the user how to vote setting up a steering committee and what's next we have just run a galga summer of code
21:05
using pg pool on the back end to try to see to improve the reliability scalability of the solution and then we started now a new european projects about water and within this project we
21:21
will introduce some more function for that analysis and time series analysis within the project thank you i'm sorry to be late maybe any questions so you said you had to
21:50
add an extension to the standard sos was it sos or was it the uh the other service the uh the sos so what was that with the extension what did you have to extend
22:02
no ways and some functionality some new parameters that were not in the standard and other things data format output data format oh those were the extensions okay yeah spiritual feature let's say yeah and one of the worst things if you have used
22:20
i think that one of the worst thing of this standard is really the observation and models the standard because it's uh it's so generics that you cannot really you can model a single observation in five different ways and so it's almost impossible to implement a client that can
22:41
read everything this is my point any more question yes it take inspiration from this
23:05
but it's not actually the implementation of this uh document that never gets to be a standard if i'm wrong which one the one that we have implemented yes we are going to document it in
23:24
the documentation not yet done but we are going to do it yes in the tutorial that probably you can download on the web there is the exercise where you can set up and shows you how it works that the principle is the same you activate the notification then the user can subscribe or
23:46
they subscribes and then provide some information how they want to be notified and then there is this uh scheduler the check and send the notification one any more question
24:04
i have a two questions um a quick one is this implementation support source two nope okay um you say you have a virtual procedure with basically doing the calculations right like uh how this virtual procedure related to the actual procedure where do you
24:25
model this there is an interface i don't know if i should there is an interface where you can feed the basically the code the process because here you see you see you can instantiate the
24:40
procedure you can get this data from these procedures and then you can you have to execute class error methods to elaborate this data and produce there so actually when i request get observation request i call this procedure not the actual procedure right
25:02
if you call this uh v grab of uh-huh yes the request goes to the server that understands this virtual procedure and call this data process and give you back the process and this is made available as part of get capability response yeah when you make a
25:20
capability you get observation all the stuff like other normal procedure okay all right okay then so that's the end of the session thank you everyone