Introducing WIS 2.0 in a box: an open source and open standards platform for international weather, climate and water data discovery, access, and visualization
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/69097 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Firenze 2022147 / 351
1
7
13
22
25
31
33
36
39
41
43
44
46
52
53
55
58
59
60
76
80
93
98
104
108
127
128
133
135
141
142
143
150
151
168
173
176
178
190
196
200
201
202
204
211
219
225
226
236
242
251
258
263
270
284
285
292
00:00
Water vaporContext awarenessDirection (geometry)Ocean currentMaß <Mathematik>Computer networkMessage passingDigital electronicsPhysical systemTelecommunicationTime evolutionArchitectureImplementationAreaService-oriented architectureLibrary catalogMetadataPoint cloudCommunications protocolOpen setWorkstation <Musikinstrument>State of matterTime domainSystem programmingFocus (optics)InternetworkingNumberRegular graphPressureCuboidProjective planePresentation of a groupCache (computing)Data exchangeMultiplication signSoftwareInternetworkingUniform boundedness principleData transmissionINTEGRALWater vaporWeb 2.0Self-organizationFreewareContext awarenessOpen setSlide rulePhysical systemWorkstation <Musikinstrument>Software developerService-oriented architectureSet (mathematics)Standard deviationMereologyEvent horizonSatelliteTelecommunicationSeries (mathematics)Point cloudState observerFrequencyWorld Wide Web ConsortiumDifferent (Kate Ryan album)Revision controlFile formatVolume (thermodynamics)Information systemsPattern recognitionDomain nameState of matterOpen sourceNumberLibrary catalogArchitecturePay televisionDigital electronicsClosed setEvent-driven programmingMessage passingDemo (music)Computer animationEngineering drawingDiagram
05:53
Service-oriented architectureFile Transfer ProtocolWorkstation <Musikinstrument>Event horizonWeb serviceFile formatBuffer solutionCASE <Informatik>MereologySoftwareTransformation (genetics)Data conversionConstraint (mathematics)MultilaterationEvent-driven programmingWeb 2.0State observerComputer animation
06:38
Set (mathematics)File formatComputer fileConfiguration spaceAnalog-to-digital converterProcess (computing)Generic programmingData conversionParameter (computer programming)Common Language InfrastructureElement (mathematics)Formal languageImplementationWorkstation <Musikinstrument>MetadataTemplate (C++)Open sourceArchitectureArtificial neural networkService-oriented architectureSoftwareComputer hardwareTransformation (genetics)SatelliteStandard deviationMathematical analysisData transmissionContinuous functionPoint cloudFreewareComputer networkMessage passingCommunications protocolElasticity (physics)ZugriffskontrolleUser interfaceAudiovisualisierungCache (computing)Link (knot theory)Uniform resource locatorCore dumpWeightCore dumpInformationInformation systemsStandard deviationTemplate (C++)Real-time operating systemOpen setFile formatBuffer solutionOpen sourceData conversionContent delivery networkSoftwareImplementationState observerResultantCuboidLevel (video gaming)MappingWorkstation <Musikinstrument>Water vaporElectric generatorMetadataMessage passingPerspective (visual)Decision theoryNetwork topologyCache (computing)Web 2.0Point cloudMultiplication signBinary fileQuicksortGraph (mathematics)Type theoryEvent horizonState of matterHookingLibrary (computing)Software developerSlide ruleComputer hardwareData exchangeMereologyGreatest elementMassArchitectureDomain nameVisualization (computer graphics)Service-oriented architectureUniform boundedness principleCASE <Informatik>World Wide Web ConsortiumConnectivity (graph theory)Projective planeFreewarePhysical systemFlagComputer configurationBinary codeCentralizer and normalizerComputer animation
13:07
Service-oriented architectureCore dumpMereologyLibrary catalogAuthenticationAuthorizationProcess (computing)SoftwarePhysical systemSynchronizationDiallyl disulfideDigital photographyMetadataPay televisionRun time (program lifecycle phase)System identificationSurfaceWorkstation <Musikinstrument>Level (video gaming)GeometryCoordinate systemUser profilePoint (geometry)Maß <Mathematik>Elasticity (physics)Standard deviationOpen sourceData storage deviceFunction (mathematics)Message passingCartesian coordinate systemData managementProduct (business)ZugriffskontrollePhase transitionData conversionCondition numberDistribution (mathematics)State of matterInternet forumRepository (publishing)Revision controlSearch engine (computing)Decision support systemCartesian coordinate systemCuboidTerm (mathematics)Uniform boundedness principleDomain nameRun time (program lifecycle phase)System identificationImplementationRight angleData exchangeWorkstation <Musikinstrument>ArchitectureMassService-oriented architectureReal-time operating systemLevel (video gaming)Vapor barrierSelf-organizationPhase transitionShared memoryFile formatSoftwareOpen setMetadataStandard deviationWater vaporAdditionConnectivity (graph theory)Image registrationCategory of beingNumbering schemePlug-in (computing)Internationalization and localizationProjective planeProduct (business)IterationUser interfaceData managementMeta elementProcess (computing)Multiplication signCore dumpData storage deviceInternet forumComputer fileHookingMereologyConfiguration spaceDialectGame controllerStack (abstract data type)Web 2.0View (database)Royal NavyComputer animationProgram flowchart
19:36
Open sourceDistribution (mathematics)State of matterCondition numberInternet forumOpen setRepository (publishing)Revision controlWater vaporData exchangePresentation of a groupMultiplication signComputer animation
Transcript: English(auto-generated)
00:01
Thank you, thank you very much. Okay, so Dave and I are here to discuss the WIST 2 in a Box project. We're going to provide some background and context as well as what led us up to creating this project and give you an overview of what it's all about, why we're working on it and what the future holds.
00:22
This is a very exciting time for the WMO and for International Weather, Climate and Water Data Exchange and we're going to tell you all about it and the exciting things that are planned with heavy integration of FOSS and FOS4G. So this is a revolutionary time and I hope you guys enjoy the presentation. I'm just going to turn it over to Dave.
00:42
Okay, thank you Tom. So we thought we'd start with some context on the WIST 2 box and background. So the WIST is the WMO information system and a system used to exchange weather data between WMO member states and use in weather forecasting amongst other things.
01:02
In looking at the context, I think it's always useful to go back in time to look at the origins of international data exchange, interoperability and things like that. And in the weather context, this can really be traced back to the 1850s where they looked at exchanging standardized weather observations over the oceans to then make weather charts, sailing charts over the oceans.
01:26
As an oceanographer, I'd like to remind the meteorological community that we were there first with the oceans and that the weather came later. And in 1873, we had the International Meteorological Organization formed to further coordinate this
01:41
exchange of data, free and open exchange, standardization, interoperability and all of those good things. And I think this helps set some of the context for what we're doing and that there's been this ecosystem developed over a long time period. And this data exchange has evolved from under the International Meteorological Organization and now under the World Meteorological Organization.
02:08
Some key dates to note are the development of the World Weather Watch. So this evolved from the start of the satellite era and truly global coverage for weather observations.
02:21
To help exchange the data and high, large volumes of data, we had the global telecommunication system implemented and developed in the 1970s. And so this was a network for exchange in weather observations prior to the internet. And this system is still in use today and it is a closed ecosystem. A simple circuit diagram there showing the message restriction circuit in its full glory.
02:46
And so in recognition that we needed to be more open with the exchange of data, we had the first demo information system developed in around 2007, starting to use web technologies, web methods. However, with changing technologies in time, we're looking at the second version of the demo information system now, looking to use
03:10
more open standards, so the OGC, W3C standards, looking at open source software, trying to make it as open as possible. Looking at data sharing through the web, using PubSub methods to advertise data, make it available to the interested users.
03:26
Making sure that what we develop is cloud ready and can be deployed either in the cloud or on premises. And then making use of web APIs to provide access to the data and services. From this work, there's been effort in specifying the architecture, which I've
03:43
tried to illustrate on this next slide, showing the concept of global services. So a global data catalog, a global data broker, global cache, feeding data to the users. And then feeding into the global services, we have a series of WIS2 nodes, which are essentially the national weather services collecting the data from their weather stations,
04:05
converting it to the required data formats for use in the operational forecasting. When those data are ready, they notify users going through the global broker that the data are ready, and the users can then go and either get the data directly from the WIS2 node or from the global cache.
04:21
And here we're talking about very large volumes of data, which is part of the reason for the global services, and it's all event driven. In developing this system, we've had a series of demonstration projects looking at different aspects. So looking at the data set discovery, how we do the catalog services, looking at
04:43
the data exchange, looking to see whether the PubSub method and subscription and download and works. We've looked at the different earth system domains, so not just the atmosphere, but the cryosphere, the oceans. And importantly, we've also been looking to support the less developed countries and small island developing states to make sure we leave no WMO member behind,
05:03
and that what we develop here works for those countries as well as the bigger, more resource-rich countries. One of these projects was working with Malawi, which I'm going to talk about now for the next couple of minutes. So looking at demonstration projects, improving the observing network and exchange of data from those stations.
05:23
So essentially, we had a system where we had a number of stations making only two or three observations per day, a few making four. This wasn't really suitable for modern weather forecasting where we need early observations ideally.
05:41
There were issues with the data transmission and things like that. And so the Malawi project was born to try and improve this system. I'm just going to skip over this next slide and move on to the solution we developed. So here, in this case, making use of Amazon Web Services, looking at the data from the weather stations coming in via SFTP in this case,
06:04
and there were constraints on the observer network that we were working with, so this had to be via FTP. As part of this, from the FTP server, we then had event-driven data transformation to the buffer format, which is expected by the weather forecasts and centers.
06:21
But we've also done a buffer to GeoJSON converter to work with the web APIs and make it more accessible to other users. And this is all then accessible through Python, Arc, QGIS, even Excel, and Tom will say more about this later. In doing this work, we've come up with several open-source software developments, the first of which is a converter to the buffer format.
06:46
I won't say much more on that today unless anyone is interested, but come and see me afterwards. More interestingly, we've also got the buffer to GeoJSON converter, so there's a lot of buffer data out there available, and if you know where to look, you can find it with the real-time weather observations.
07:03
But it can be difficult to use, so we've got our buffer to GeoJSON converter. This is also a requirement for the open APIs and web technologies we're using, and it should be able to work with most, nearly all, observational buffer data to convert to GeoJSON. We've published this all on GitHub under the Apache 2 license,
07:24
and I should flag that we make heavy use of the EC codes library from ECMWF. A quick illustration of the implementation. So we have our CSVs coming in, station metadata, giving contextual information on height of sensors, types of sensors, a template to map to the buffer format, the conversion to buffer, user unfriendly binary format,
07:47
and then conversion to GeoJSON for use in the WIS2 box. And overall, this has improved the access to the observations and availability of those data from Malawi, and this work has then turned into the WIS2 box, opening up access to many other users.
08:03
And with that, I'll pass to Tom. Thanks, Dave. So how do you move forward based on a point-to-point, private, bespoke networking system and try to modernize international weather, climate, and water data exchange?
08:24
The answer to that is working more in the open, and that means open standards and reference implementations of open standards using free and open source software. So I will quote Cliff Cottman, interoperability doesn't happen by accident. And that's very true, and that's something we've really taken to heart in this generation of WMO data exchange.
08:46
So I'm just recanting some of the history that Dave mentioned that led up to the inception of the WIS2 box project which started in November for a reference implementation for data exchange.
09:01
And that's available, free, open source, we want it available for all members. So what is WIS2 in a box? It's a reference implementation of a WIS2 node. So a WIS2 node is a specific component of the architecture we're developing as part of the next generation WMO information system. The core components of WIS2 box are PubSub, or MQTT for that matter in this case.
09:28
Weather data does not happen on a cron. It's a very event-driven domain, so we are making heavy use of published and subscribed technologies, and obviously HTTP.
09:40
WIS2 in a box is software, it's not hardware. You can't buy the box, it's software, you install it, and then it works either on premises or on the cloud. So it's basically a reference implementation of that part of the architecture. The things that we do with WIS2 box, so it's plug and play, not plug and pray.
10:03
It's based on international standards, heavy use of OGC standards here, and that's built in from the bottom, or from the start, as well as ISO and W3C standards for that matter. Again, on premises or cloud capabilities. We have a data exchange capability in WIS2 box, so you install the box,
10:25
you hook up your real-time data feed, or your non-real-time data feed, and it sort of runs through the box. There's an ingest, and I'll walk through some of the workflow later, but data exchange is a big piece. Data visualization is a big piece, looking at data through graphs or maps.
10:45
And weather data is very, weather, climate, and water data are very exciting, but I don't have to tell you guys, when you put data on a map, something happens. And that's what is happening here in this next generation of WMO and Phosphor-G, for that matter, and geospatial.
11:01
Discovery is a big piece, obviously, both discovery from a user perspective in the community, as well as a mass market perspective, as well as monitoring. We are totally open, again, based on open source. I can't show this slide enough, as well as free and open source standards. So you'll see some of the usual Phosphor-G suspects on the free and open source software slide,
11:22
and we're pretty happy about that, and there's nothing proprietary here. Everything is out in the open, and that's how we choose to operate and make things available. So open source implementation of open standards, of which WIST2 is based on these open principles.
11:41
In fact, being open is a principle of WIST2 that was defined a few years back. In the WIST2 architecture, we have the idea of core services and compliance services. So core are the things you shall do as a member state, as a member, and the optional things are things you should do or may do, for that matter.
12:01
So the core services, again, are PubSub, using the web and using both of those to either subscribe to data, download data, or make your data available through events to the infrastructure. The optional services are APIs, and we allow OGC APIs as well as other well-defined APIs,
12:22
and we make heavy use of that in WIST2 Box as a result of that decision from the architecture team at WMO. Simple workflow, the whole idea is in WMO, we have, WMO has also revamped their data policy, which forces members to provide data in a more openly available manner for what we classify as core data.
12:48
So with the core, all WMO core data shall be free and unrestricted, and that is driving a lot of the WIST2 work. So the whole idea there is that core data is made available to global services,
13:00
and those global services from those central services, such as we'll have global broker services, global caches and CDNs of real-time weather and climate and water data will be made available through those global services for anyone, whether you're in the weather community or whether you want to build a weather application.
13:20
Again, the unified data policy is pushing a lot of that. We also have a core capability in the unified data policy, and we also define recommended data, because we do realize some data is not designed to be open. So for that data, there is an idea of access control in the architecture,
13:45
and we make that a reality in WIST2 Box itself. So there's a lot of flexibility. The other part where WIST2 Box helps as an implementation is it could support national or regional data exchange. So in addition to providing your data to WMO as an agency, you can also set up WIST2 Box
14:06
and do regional data sharing if you wanted to do more local data sharing in addition to going out to WMO Global Services. Here's what the whole thing looks like from an architecture perspective.
14:21
So WIST2 Box is obviously in the navy blue, and it's designed to, again, publish, subscribe, support, publish, subscribe, and download of weather and climate water data. And it's mass-market friendly, so it'll work with search engines, it'll work with GIS applications.
14:40
It'll also work with the decision support tools that we're familiar with in the domain, such as forecasts of workstations and so on. So not only are we trying to satisfy the domain, but we actually want to fan out and serve the mass market and maybe the people beyond our community to reach out to them and make better use of this data.
15:01
We have two concepts in WIST2 Box. We have a design time concept and a run time concept. So when you install the box, you basically run a couple of design steps to set up your metadata and hook up your data feeds and configurations, and that you can see in the blue. And everything after that is run time, which means you don't touch it. So you configure it, you set it, maybe you don't forget it, but you let it run,
15:24
and it does its thing, and the engine turns, and then it contributes to the data exchange. What happens? I'll walk you through an actual workflow. So a data file will arrive in WIST2 Box, we identify the data, so there's an idea of a topic category in the WIST2 architecture,
15:45
which is a very deep classification scheme on what kind of data this is and who it's coming from and so on. So that goes through an identification step. We have a data pipeline step, which is extensible. So all of WIST2 Box is built on a plug-in architecture.
16:02
So organizations may want to implement their own data pipelines to deal with their own data formats and basically write those and integrate those into the box. After data runs through the pipeline, it's published to a web accessible folder, which then a notification is made using PubSub and MQTT to WIST2 global infrastructure.
16:25
We also publish data to the API, so that gets made available through an OGC API. Then the user on the right will subscribe to the PubSub and get a notification back that the data is actually there and do what they have to. So a few screenshots. Here's an early iteration of what we're working on.
16:45
So a very basic map. This is all driven by OGC API features and other standards. Here you can see some of the GeoJSON that Dave was referring to. So we obviously support that through the OGC API.
17:02
You can use this in QGIS. You can even use it in Excel for better or for worse. But the idea here is that we've lowered the barrier to access the data using the open standards. Phosphor-G. Phosphor-G is sprinkled in many places on the project.
17:21
So here's our software stack. You'll see use of PyGeo API, PyGeo meta, OWS lib. We've discussed some of these during the conference. So they've been useful in helping us set up this entire box, as well as some other technologies. There are leaflets there as well for the user interface. Here's a look at the architecture.
17:41
Basically we have a data management component which basically does the identification and the ingest and the processing and the registration. Everything is S3 storage. We're using MinIO. We have a PyGeo API application which is container which is doing the API stuff. We have a light view JS and leaflet capability.
18:02
And then we're using Mosquito for the PubSub. And there we can see how Phosphor-G has infiltrated the Wis2Box capabilities supporting the open standards. So we're pretty excited about the project in and of itself in terms of where WMO is going with regards
18:21
to being more open in terms of data exchange and the idea of having a reference implementation to support the architecture. So a user can come and say, I want to see what this actually looks like if I implement it. And we can say, here, it's right here. Go download it and it's Python. And if I can read it, anybody can read it. So there's something to be said for that.
18:41
In terms of a roadmap, we started in November. We're looking at a 1.0 by the end of the calendar year. In WMO, we're looking at a pilot phase later on in this fall of a year between various members to test the architecture, to validate the architecture. And in 2024, we're looking at moving some of these things into production.
19:01
So it is an agile, rapid, and iterative approach, which is really nice and positive to see. And Wis2Box is moving along with the rest of the WMO architecture to provide this out-of-the-box component to make this all happen. Again, totally open. Everything is on GitHub.
19:21
It is an Apache 2 license. We have a discussion issue tracker. There's a forum there for people to want to have discussions and so on. And we have our official docs at docs.wis2box.wis.wmo.int. With that, we're going to stop. I'd like to thank everybody for their interest in this presentation.
19:43
Again, this is a really exciting time for WMO and data exchange, and we're happy that we were able to present this to you today and here this week.