jsMapservice: leveraging the open source stack for rapid publishing of spatial data
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 52 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44692 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSS4G SotM Oceania 201938 / 52
9
11
12
14
15
17
20
23
26
28
30
32
34
39
44
00:00
Open sourceElectronic meeting systemTable (information)Variety (linguistics)Service (economics)Configuration spaceContent (media)Group actionFunctional (mathematics)Local ringQuery languageTable (information)Communications protocolClient (computing)Query languageMultiplication signTheory of relativityCodeView (database)Right angleLevel (video gaming)Sampling (statistics)Scaling (geometry)Cursor (computers)MassVisualization (computer graphics)BitWeb applicationGame controllerRoundness (object)Open sourceDatabaseConfiguration spaceSoftware testingService (economics)MappingGroup actionScripting languageVariety (linguistics)MultiplicationRepository (publishing)Computer filePolygonMachine visionPoint (geometry)Projective planeRow (database)Archaeological field surveyRevision controlResultantEquivalence relationElectronic visual displayTelecommunicationCoordinate systemMereologyWordWeb 2.0Content (media)Axiom of choiceAttribute grammarComputer programmingState of matterVideo gameSet (mathematics)Cartesian coordinate systemServer (computing)
05:10
Image registrationScalable Coherent InterfaceMetadataData managementUsabilityArchitectureTemplate (C++)Service (economics)Keyboard shortcutComputer networkNP-hardOpen sourceMathematical analysisRevision controlBridging (networking)CodeJava appletExistenceMetadataScripting languageView (database)Configuration spaceService (economics)Library (computing)Process (computing)Client (computing)CASE <Informatik>GeometryComputer configurationComputer fileSoftwareRevision controlMIDILevel (video gaming)Virtuelles privates NetzwerkData managementOpen sourceMereologyWeb pageBridging (networking)Link (knot theory)DatabaseUniform resource locatorFile viewerObject (grammar)Group actionTable (information)Suite (music)Dean numberSpherical capDataflowQuery languageVideo gameObject-oriented programmingMappingNumberFreewarePoint (geometry)Resultant10 (number)Greatest elementSoftware frameworkRight angleRepresentational state transferComputer architectureGoodness of fitMobile appSystem administratorCartesian coordinate systemDecision theoryWeb serviceMoment (mathematics)Demo (music)Server (computing)Interface (computing)Template (C++)Form (programming)Replication (computing)Content (media)Product (business)InternetworkingAuthorizationHyperlinkPhysical systemMixture modelInteractive televisionWeb-DesignerCuboid1 (number)Internet service providerImage registrationElectronic mailing listField (computer science)AbstractionÄquivalenzprinzip <Physik>Existential quantificationSubject indexingFile formatSoftware testingEndliche ModelltheorieArtistic renderingJava appletCodeTranslation (relic)Medical imagingBitCentralizer and normalizerAttribute grammarDefault (computer science)Axiom of choiceSource codeQuicksortTelecommunicationWeb 2.0Schmelze <Betrieb>Multiplication signPlug-in (computing)Social classGame controllerElectronic visual displayMathematical analysisLoginFront and back endsHydraulic jump
Transcript: English(auto-generated)
00:02
Hi, I'm from Genius Science, which for people that aren't in New Zealand, would be the equivalent of the geological surveys in most part of the world. And the word survey would probably give you the idea that we've actually got quite a lot of spatial data that we want to make available to public, to our stakeholders,
00:21
and also internally to the rest of the science, to first get into communication with the science groups. We have evolved. JS Map Service is one way of doing this, and has made fairly heavy use of the open source stack to do that, and I just want to talk a little bit about how
00:45
we did it, what our learnings were, what are the choices that we made. Coming back to the problem that I'm trying to solve, which goes back quite a while, is first off that we're wanting maps that can consume a wide variety of web map services.
01:04
Aggregation with other people, NOAA, DOC, other government agencies is incredibly important rather than copying it after all the time, but we don't have any role in saying what protocol they're going to use. A biggie was that we wanted the content to be configuration-driven.
01:23
Life's too short to be building a web application for every single map that we want to put out there. And very important associated with that is the view that someone sees depends on what rights they have to the data. We wanted to put the science groups in control of the data, which means putting appropriate
01:42
controls around testing and the final push the button to make it live on the web, but it has to be simple enough and easy enough for scientists to be able to manage. Version 1, version 2 were configuration file-driven. We jumped to database for doing it, for the latest version.
02:05
We wanted highly functional maps, not minimal maps. We want maps where you can do pretty much a lot of the things that you do with a GIS. We wanted scientists to be able to add and remove layers, to be able to share what they're seeing with other people while just passing the URL.
02:23
And the biggie was that we needed rapid publishing from projects which had hundreds of layers and making up a map. Very important too was obviously, we're not doing this just for show, we're dealing with data sets that aren't useful until you start doing querying on them and we wanted
02:44
to support pretty high level of query. Most of the tickets that are sitting in the database are around querying and visualization. And particularly, we wanted to be able to support queries based on not just the data that you're seeing on the screen, but usually they have a mass of related tables
03:05
which can come from multiple data sources, which I'll just illustrate. You may have your client which has got navigation which might be available throughout GIS server with a definitive note that you're sitting at NOAA.
03:25
Samples from this cruise might be sitting in a sample repository, e.g. scripts which might be using WFS or pseudo-WFS where they still use FS but there's no spatial.
03:42
And because you were on that cruise, you might have private unpublished data that's still sitting in the corporate database. And we wanted queries so that hide all of that from the users and allow them to use all of those available sources for querying and display.
04:01
An end result of a map is something that looks a bit like this. So this is from the Atlas of Petroleum Prospectivity. We've got the usual kinds of search by, you know, just touch a point, polygon searches, search by attributes, and searching by just a text search.
04:27
All the menus at the top are also part of the configuration. You just set them up and you can see some idea of just how many layers are sitting in this map, about 60.
04:41
When you query, you get results that you can customize, export to CSV for doing other things with. We're busy looking at how we can change those exports to put them out to a Python data lens. There's another way of giving people quick access to results.
05:01
And all the usual things that you expect on maps to be able to see where your cursor is in any of the coordinate systems, scale, and all the other usual bits and pieces. I don't want to go spend any time really talking about the map applications. We have quite a number, some of them are public facing. The most well known is Petroleum Basin Explorer.
05:26
Tens of maps, many hundred layers, thousands of documents, but it's free registration to get into it. But more recently we've got the Antarctic Geomap and the Zealandia map,
05:42
which are both public, almost all the data is basically public and readily available if you want to look at those. They're quite small at the moment, expect Zealandia to explode in the amount of data content over the next few years. What I do want to talk about is how we made this possible and the architecture.
06:04
So this is broadly the architecture where we start with your clients, which can be the full blown one with the menus all pre-configured. But they could also be embedded just to be a map or map plus legend embedded in another HTML page.
06:26
Everything they do is going to be fetching stuff from the map service through the REST API. And also they will be fetching JavaScript and HTML templates directly from that.
06:41
The advantage for us is that by doing a running, pushing an update through to map service, we're effectively updating all of those clients simultaneously. The clients will then be requesting particular maps or whatever, and they will go down to our database where adjacent objects will return a view appropriate to their rights back to the client,
07:09
which then is going out to the various web services out on the web to go and actually display the data. We've got a central authorisation server to manage rights and of course access to corporate database.
07:25
And we have a G network server to support associated metadata and a full text indexer to provide for full text searches.
07:42
And of course right at the bottom we have the administrator, which runs an associated app which builds, manages and builds all of those. So with appropriate admin rights our users, a group of science groups can go in and fill all the content, add more layers, control behaviours, etc.
08:05
Going into more detail about the open source choices, because we did drive this on open source, the last big rewrite was 2016 and so architectural choices that we made there date back to then. Maybe we'll make different ones now. Client wise we jumped for Angular for many reasons,
08:22
but certainly the template, the ability to instantly include templates were great. Leaflet.js, version 1.2 were open layers, but we jumped to Leaflet because of its nice plugin architecture. It supported our target services pretty much straight out of the box.
08:40
And in particular vast amounts of people, as you are well aware, are publishing their data through ArcGIS server and we wanted basically first class access to that data. And then Leaflet, their ESRI plugin is very well supported.
09:00
And wrapping all this inside our map service binding which abstracts the layers and abstracts the queries which allows us so that if we've got some new kind of beastie that we want to plug in then it's relatively straightforward just to write the appropriate methods to do it.
09:23
Backend. We've got ArcGIS server, but for most of the stuff that's involved in this kind of publishing the stuff is going out to GIS server of Postgres. We could have an entire conference on the 18 meridian.
09:42
A certain GIS provider assured us in 1987 that they'd have problems fixed in the next version. I think the story is still the same. But the advantage of the open source was that we could fix the problem ourselves by just ripping into the source code and pushing in a pull request
10:02
or we could actually just go and pay someone else or the lead developer to go in and give us a spherical earth, thank you very much. And we've done both. Performance was fine with thousands of layers whereas our ArcGIS server manager gets a little leery when you say all there's 600 layers here.
10:23
And of course we appreciate the price for Postgres as opposed to Oracle Spatial. Apache Solr is what we use as our full text engine and I have to say that we just use it because that's what we've always used. Haven't seen a good reason to change it.
10:41
H2 is what we use for our embedded database. It's embedded which is what we want to perform it. I suspect that if I'd made the decision today knowing how this thing has evolved, I'd have gone for Mongo and there's every chance of version 4 of this we will make the jump to Mongo
11:03
something we're actively looking at. Geonetwork, well it suited our management because it allowed a tightly controlled management of the publishing workflow and from my point of view it's very easy to provide links to the metadata and the backward links for searching.
11:21
If the layers found through Geonetwork happily jumps to references into our GeoServer and into our applications. I'm not brave enough to try doing live demo of our admin interface navigating the VPNs necessary to get to work.
11:43
These are just screenshots just to give you a flavour of how the configuration is operating. This is just a view of a library all of its layers organised in a hopefully easy way and this is an example of a configuration of a layer
12:00
which in this case is a primary just coming in through WMS and we just set up the usual bits and pieces about how the thing's going to be managed for legend you can set up much more complicated legend options with a JSON descriptor there if need be links to URLs for downloads and metadata
12:28
and just how tooltips work how different styles that you might view the data are set up metadata links if there was a 3D version there would be a URL that fired up a 3D model
12:41
to look at it override the SLD for the rendering and this is just setting up the fields for display which is all pretty straightforward we control who can see that field
13:02
how it's presented to the user whether it's available for attribute-based search, text-based search a whole syntax on how to visualise that primarily for creating hyperlinks to the system and access to that link as well
13:23
and usual hints but up the top here there's also this one is indicating this one has got a data viewer a data viewer is an object that we create for managing linked tables and I'll show you a bit about how that's configured shortly
13:42
and for creating a map then it's pretty much choose a library find a layer override some of its defaults add it to the map and over on this side arrange your layers and groups and so forth however you want this thing to be presented to the user
14:03
how do we deal with data that's sitting are they related tables this is an example of a data viewer over here are all of the different tables that are associated with the spatial object in this case they are all JDBC links
14:21
but if they were different then we can set up an individual connector the basics of this is obviously how do we select it which will obviously be different depending for JDBC or WPS or AGS which are the three that we support
14:41
and very similar to the rest of it how we can display the fields if something's got a data viewer associated with it then we are able to see all of these set up and do form based filtering to set up the query and you get the results back in a data viewer like this
15:03
which again allows you to look at the individual data look at the individual data and potentially export it all out to Excel for further analysis the final bit I really want to talk about
15:22
is the publishing workflow because how do we do this hundreds of layers and here we want to sing out to a product called Geocap Bridge which is not open source not free absolutely incredibly high value take an ArcGIS MXD file and it will move data to Postgres
15:40
publish the layers including translating the styles into SLDs to GeoServer push the metadata through to Geonetwork all with one button in practice you might want to control that process a little bit more but for all that you can put out one of these things incredibly quick and then we have some Java code
16:01
which analyzes the MXD emits a JSON that the thing can configure for putting up for the purposes and we're looking at doing a QIS version Geocap Bridge is also looking at doing a QGIS version it's worth it to my mind
16:21
simply for the Geocap Bridge to my mind is worth it simply for the translation of styles the unsung hero would be the Serenity testing framework that we're very late comers too but now that I've got into it my goodness it makes life easier with Cucumber
16:42
and the oil in the wheels that make all of this possible of course is an exposed API and so the sheer advantage of building things with an API allows all of this kind of fun to happen. Thank you
17:09
Thanks Phil. Are there any questions for Phil? We've got quite a few minutes Yep, one up there
17:27
Hello I'm curious you said you're sending queries to multiple databases so how do you split queries
17:41
up between them? How about badly? We take a rough guess as to what's going to be the most efficient way to do it and it forms a series of inner joins as it comes back I would not say this was
18:00
the most performant part of our application but on the other hand compared to doing it by any other method it gets you the data incredibly fast Ah cool, yeah I understand
18:21
Phil I'm interested in how the metadata flows across it. One of the amazing things is how often I download data and no metadata comes with it, particularly the publishing date and sources so if I download some data I don't get an XML file with it normally
18:43
Metadata is a case of sitting there with a large whip People scientists on the whole want their data out there for their users for their public and I pretty much say yes when you've got the metadata The
19:01
normal flow for the data is that they would create that in whatever tool suits them I prefer the GIS one but whatever floats your boat by and large We pull them down as MEF files, we have Python scripts that run over, dolly them up
19:21
bring in links to the GeoServer images and then do a big bulk upload but of course Geocap Bridge if there's metadata available for the data we'll do all that for you Good time for probably one more
19:53
I'm interested in the fact to you about how you engage with the scientists Now scientists aren't the best communicators so
20:01
I'm just wondering how you manage turning what the scientists might want to put in as label names and layer names into something that's meaningful to the users There's a real mixture of that quite a lot of this is user driven They're telling us what they want
20:23
and besides from that there's the usual interactions that go on internal review Me looking at it showing it to the selected users with a login and then final publishing A lot of the formats and so forth
20:41
are coming through because users want them This is supplying what they need