Exploring the World’s Open Data Portals - Discovery, Visualisation and Analysis of Open Data with TerriaJS
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 351 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68993 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Year | 2022 |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Open setSmith chartMathematical analysisVisualization (computer graphics)Open setSelf-organizationComputer animation
00:22
Web browserSoftware frameworkLibrary catalogSource codeWebsiteSoftware frameworkWeb applicationOpen sourceLibrary catalogWeb browserComputer animation
00:40
Workstation <Musikinstrument>GUI widgetLibrary catalogNetwork topologyData structureMetadataRational numberLibrary catalogMetadataData structureConnectivity (graph theory)Network topologyTexture mappingSource codeGame controllerMultiplication signComputer animation
01:07
Vector spaceData typeScripting languageOpen sourceConfiguration spaceNetwork topologyTwin primeConnectivity (graph theory)10 (number)File formatGame controllerMathematicsVolumenvisualisierungMappingVector spaceDigitizingTesselationEnterprise architectureOpen setCodeClosed setFocus (optics)Different (Kate Ryan album)Texture mappingSoftwareWeb browserClient (computing)Data conversionType theoryComputer animation
02:02
Visualization (computer graphics)Computing platformMoment (mathematics)Proxy serverTexture mappingVisualization (computer graphics)Set (mathematics)Computer animation
02:19
Cone penetration testOpen setDigital signalSatelliteProduct (business)Mathematical analysisOpen setSet (mathematics)MappingSatelliteWeb serviceCubeDigitizingProduct (business)Computer animation
02:49
Open setCone penetration testPersonal digital assistantDigital signalTwin primeMultilaterationBitTexture mappingConditional-access moduleClosed setState of matterDigitizingTwin primeComputer animation
03:12
Asynchronous Transfer ModeSet (mathematics)Type theoryMeasurementDigital signalTwin primePressureTexture mappingBitTesselationLevel (video gaming)Image resolutionComputer animation
03:42
Web browserProxy serverVector spaceTable (information)Open setService (economics)Client (computing)Raster graphicsLimit (category theory)Presentation of a groupTable (information)Mathematical analysisFile formatProxy serverVideoconferencingWeb browserVisualization (computer graphics)Source codeSheaf (mathematics)Heegaard splittingPoint (geometry)Connected spaceMappingTesselationStatisticsWeb 2.0Web serviceTexture mappingOpen setRaster graphicsLibrary catalogLibrary (computing)Revision controlEquivalence relationProper mapGeometryComputer reservations systemTouchscreenComplex analysisConditional-access moduleDifferent (Kate Ryan album)State observerSelf-organizationSet (mathematics)Service (economics)Software engineeringVector spaceComputer fileComputer animation
06:05
Set (mathematics)Library catalogIntegrated development environmentFood energyData structureNetwork topologyGroup actionOpen setService (economics)Query languageLocal GroupImplementationBeta functionComponent-based software engineeringAerodynamicsPrice indexFluid staticsPlastikkarteCountingMetadataLink (knot theory)Uniform resource locatorSubject indexingWeb browserNetwork topologyMetadataSet (mathematics)TouchscreenGroup actionOpen setData structureSubject indexingType theoryDynamical systemQuery languageImplementationComputer fileLibrary catalogServer (computing)Web browserVisualization (computer graphics)Source codeBitDirectory serviceService (economics)Computer animation
07:10
Texture mappingVector spaceOpen setVisualization (computer graphics)Table (information)Series (mathematics)Case moddingCovering spaceTotal S.A.Set (mathematics)Web serviceTexture mappingSpacetimeDimensional analysisSet (mathematics)Process (computing)Time seriesCovering spaceVisualization (computer graphics)Different (Kate Ryan album)TouchscreenHeegaard splittingSelectivity (electronic)Multiplication signGoodness of fitMathematicsComputer animation
08:06
Cross-site scriptingVector spaceBeta functionSmith chartRouter (computing)WebsiteActive contour modelGAUSS (software)Real-time operating systemWeb servicePoint (geometry)Vector spaceGraph coloringClique-widthCategory of beingMathematicsComputer fileActive contour modelVisualization (computer graphics)Drop (liquid)SequenceScaling (geometry)Line (geometry)Attribute grammarSource codeTrigonometric functionsComputer animationSource code
09:19
Computer fontActive contour modelDesign of experimentsVector spaceTable (information)Series (mathematics)Set (mathematics)Electronic visual displayVariable (mathematics)Function (mathematics)Maxima and minimaChannel capacityType theoryTime seriesVisualization (computer graphics)Table (information)Drop (liquid)Point (geometry)Greatest elementMultiplication signUniform resource locatorOpen setPower (physics)Pairwise comparisonQuicksortServer (computing)Medical imagingComputer iconVideoconferencingMathematicsAugmented realitySource codeWorkstation <Musikinstrument>Different (Kate Ryan album)Graph coloringProcess (computing)BitTexture mappingComputer animationSource code
11:10
Table (information)Texture mappingVector spaceGeometrySet (mathematics)Boundary value problemDifferent (Kate Ryan album)Vector spaceTable (information)Texture mappingSet (mathematics)TesselationGeometryComputer animation
11:29
AreaStatisticsLevel (video gaming)Zoom lensAbsolute valueTable (information)Texture mappingEstimationEnterprise resource planningSet (mathematics)Internationalization and localizationOpen setTexture mappingMultiplication signPairwise comparisonDimensional analysisTime seriesMathematical analysisStatisticsDialectComputer animation
11:58
Digital filterProcess (computing)Web servicePairwise comparisonOpen setMathematical analysisMeasurementLine (geometry)Vector spaceTable (information)CubeCubeQuicksortProcess (computing)Web 2.0Mathematical analysisLine (geometry)TouchscreenTable (information)Open setPlug-in (computing)Latent heatPairwise comparisonSpacetimeRaw image formatSet (mathematics)Vector spaceRaster graphicsWeb serviceSoftware developerMeasurementDifferent (Kate Ryan album)Real numberComputer animation
13:08
QuicksortBoundary value problemCovering spaceLibrary catalogFraction (mathematics)Negative predictive valueTotal S.A.Software bugOverlay-NetzPolygonExistenceProcess (computing)Arithmetic meanChannel capacityWeb serviceFraction (mathematics)Web 2.0Arithmetic meanImplementationProcess (computing)Covering spaceComputer animation
13:26
Fraction (mathematics)Mobile appWeb serviceOverlay-NetzPolygonArithmetic meanSmith chartRange (statistics)PixelFunction (mathematics)Total S.A.Covering spaceTouchscreenGraph (mathematics)Process (computing)Mathematical analysisPoint (geometry)ExistenceRandelemente-MethodeTetraederGreen's functionPlastikkarteSet (mathematics)Texture mappingDialectProcess (computing)Field (computer science)PolygonOverlay-NetzWeb serviceTime seriesServer (computing)Computer animation
13:54
Open setLink (knot theory)Embedded systemLink (knot theory)State of matterMotion captureProxy serverTexture mappingUniqueness quantificationMereologyComputer animation
14:11
Link (knot theory)Uniform resource locatorDean numberFunction (mathematics)State of matterTexture mappingWebsiteUniform resource locatorFunktionalanalysisComputer animation
14:34
Link (knot theory)DemosceneCASE <Informatik>Covering spaceRobotDemosceneProxy serverLink (knot theory)Covering spaceComputer animation
14:54
Link (knot theory)DemosceneCovering spaceTotal S.A.Negative predictive valueFraction (mathematics)Software bugPeer-to-peerEvolutionarily stable strategyMotion captureSet (mathematics)Digital photographyRobotMathematicsFraction (mathematics)BitCovering spaceDemosceneComputer animation
15:27
Set (mathematics)Gamma functionNegative predictive valueCovering spaceLink (knot theory)DemosceneDrag (physics)Configuration spaceOpen setProxy serverConfiguration spaceDifferent (Kate Ryan album)Open setTexture mappingCodeMappingComputer fileOpen sourceDrop (liquid)Computer animation
15:59
Open setConfiguration spaceSet (mathematics)Drag (physics)Local ringTouchscreenInternetworkingDiallyl disulfideUniform resource locatorOpen setTexture mappingMetadataServer (computing)Different (Kate Ryan album)Client (computing)Type theoryInternet service providerRandom matrixComputer animation
16:45
Open setConfiguration spaceBoss CorporationElectronic visual displayVariable (mathematics)Set (mathematics)Game theoryOpen setBitUniform resource locatorMappingView (database)Texture mappingContext awarenessScalar fieldComputer animation
17:17
TouchscreenFunction (mathematics)Computer fileDrag (physics)Network topologySet (mathematics)Variable (mathematics)DeterminantPolygonAreaPC CardNetwork topologyDrop (liquid)TouchscreenState of matterComputer fileVideoconferencingAreaElectronic mailing listReal-time operating systemMoment (mathematics)Category of beingComputer animation
17:52
Commercial Orbital Transportation ServicesInformationVariable (mathematics)Computer fileDrag (physics)Set (mathematics)Gamma functionComputer fontVisualization (computer graphics)BitNumbering schemeMultiplication signNetwork topologyGraph coloringComputer fontComputer animation
18:22
Workstation <Musikinstrument>Configuration spaceFunction (mathematics)Complex analysisData typeScripting languageTexture mappingVisualization (computer graphics)Pairwise comparisonDiffuser (automotive)Web browserMobile WebMobile appWeb serviceTexture mappingConfiguration spaceComputer fileOpen setLibrary catalogType theoryComplex analysisCartesian coordinate systemSet (mathematics)File formatSoftware developerPhysical systemPlug-in (computing)PlanningOpen sourceProjective planeFunktionalanalysisService (economics)Uniform resource locatorComputer animation
Transcript: English(auto-generated)
00:00
Hello, my name is Nick. I'm here to talk to you about exploring the world's open data portals with Teri.js. I work for CSIRO, which is Australia's national science agency, one of the world's largest multidisciplinary science and technology organisations. If you want to know more, go to csiro.au.
00:23
Teri.js, what is it? Teri.js is an open source framework for web-based geospatial catalogue explorers. It's a cesium leaflet based in-browser tool for looking at 2D and 3D data. Website teri.io if you'd like to know more.
00:42
Teri.js has three main components. You've got a catalogue, tree structure, you can search through it, you can view metadata. Map component, quite important. 3D map, cesium 2D fallback as well. And we've also got data controls. So we've got a workbench on the left showing a legend, a little time component.
01:03
We also have a few other panels and tools that let you interact with data sources. So how do we do this? Well, open source software, leaflet, cesium, react, mobx, protomaps for the vector tile renderer. Lots and lots more usual JavaScript dependency tree of tens of thousands of things.
01:23
So our maps. We kind of make two different types of maps. We've got our fully open maps, very much open data portal focus, very much client-side running in the browser, very little server-side component. All using the same open source code base, just configuration changes.
01:40
We also do partially closed digital twin maps, 3D data focus, quite server-side heavy. More closed proprietary formats, format conversions, dealing with enterprise auth, access control. I'm not going to really go into that because it's not really in the, I don't know, it's not very open.
02:03
So our main flagship map is the Australian national map. It's the birth of Terrier JS, a platform for geospatial data discovery, visualization and sharing. Launched in 2014, got about 14,000 datasets on there at the moment. Getting about 30,000 sessions a month. When you break down those 15,000 datasets, you've got 46 open data portals that it connects to directly.
02:26
Then we've also got about 1,100 curated datasets on top of that. Another one of our, another two of our open maps are Digital Earth, Australia and Africa. So these were developed in partnership with Geoscience Australia for looking at Landsat central,
02:41
satellite imagery and derived products. And we connect directly to the open data cube, open web services stuff. Another map, Pacific Map, which was developed with the secretariat of the Pacific community, connects to the Pacific Data Hub through their CCAN and SDMX API, which I'll talk a little bit about later.
03:03
One of our closed maps, Victoria Digital Twin, is done with the state of Victoria, Australia. Just launched, lots of very cool data on there. One example is this 3D aerial imagery of the Great Ocean Road, south coast of Australia. I've forgotten the resolution, but it's very nice.
03:22
And so we've got this tool where you can kind of click on a spot in the map and it will move the camera down to the ground. You can then inspect the imagery. So it's a little bit slow loading here, but you can see as I kind of move the camera around, yes, you're getting more levels of detail popping in. So this is served through Cesium 3D tiles.
03:43
So what kind of data sources can you connect to? Again, on a web browser, one kind of caveat there is sometimes a proxy is required depending on cross origin resource sharing. Some services don't let you connect to it through a web browser, which is obviously fair enough. It's there for a reason.
04:03
So sometimes you do have to set up a proxy to allow this stuff to happen. So imagery tile services, we don't currently have proper raster support, but it is coming. So you've got your usual WMS, WMTS, Mapbox style, the ArcGIS equivalents, vector data, web feature service, ArcGIS, your usual vector file types,
04:26
and Mapbox vector tiles now. 3D sources, very much Cesium heavy, but we did just add the web assembly version of the open asset importer library, which allows you to convert about 40 different 3D file formats to glTF, so you can view them in the browser.
04:42
Very cool. Tabular sets of data, so that's point, lat, long, source, and also region mapping data. So you've got CSV, STMX is a statistical API, which is used by a lot of big organizations around the world to publish their statistical data. OGC sets observation service, Socrata, and Open Data Soft is up and coming.
05:04
Open Data Portals, so these create the data sources that I just previously outlined. So you've got CCAM, catalog service for the web, Socrata, Open Data Soft, a few others there. Again, all in a web browser. You've got your usual limitations there. File size, complexity of the geometry, what are the formats,
05:23
how can you access them. CRS, we all know about. What do I do? So I live in Sydney, software engineer for Teri.js. I mainly work on the 2D data visualization and also these portal connections and getting them to work. So, how do you explore the Open Data for Teri.js?
05:42
So I've split it up into four different sections. You've got discovery side, visualization, analysis, and then finally creating and sharing apps with people. So warning, I do have a lot of videos, very visual presentations. I hope you can see the screen. Obviously, we know about the limitations with the presentations here.
06:01
So this is the format I've tried to use. So we start with discovery of the open data. So catalog, as I pointed out earlier, tree structure, you click through like a folder directory, and as you click through certain things, metadata is going to appear depending on the sources. So these groups can be manually created for curated data sets
06:21
or if you have an open data portal, it will just appear kind of in the tree and you can browse through it. There's also the ability to query certain types of open data services. Again, it follows the tree structure. So the example on the screen is an open data soft server by a locality in Australia.
06:40
As you can see, I can browse the data by keyword and a few other things, but it just depends on the API and our implementation of connecting to that API. We've also got search, very important. So we've got two methods of searching, a dynamic in-browser search, which gets a bit complicated if you've got 46 data portals to try and search.
07:03
It's not really doable. So we've also got a tool that generates a little catalogue index file that massively speeds up searching. Visualization, the interesting stuff. So imagery, so your usual kind of imagery map visualization. This is on a 3D globe. You've got opacity, all that good stuff.
07:22
It's a time series data set. So Terrier will automatically recognize that this WMS has a time dimension. It will render a time picker. You can go through, select year, date, so on. It will also let you animate the time series. So depending on the service and the latency,
07:40
it will try and do a good job. Sometimes it succeeds, sometimes it doesn't. We've also got a compare feature, which lets you duplicate a data set and change dimensions and split it across screen space. So here I've split this WMS, and I've set the date different on the left side. So you can then drag the slider across
08:00
to see differences. So this is a vegetation cover data set. Vector data. So Terrier has an automatic vector styling engine. So the whole point of it is you can kind of drop in any vector data, and it will try and create a visualization that is somewhat sensible. So this is a 200 megabyte GNJSON file
08:22
that's come through WFS through a CCAN service in Australia of contours. And so it's recognized that there's this altitude attribute, and it's given at a color scale. But if you don't like the vector styling, we've also just added the ability to edit the style.
08:41
So here you've got your usual D3 color scales, which are coming from ColorBrewer and a few other sources. So here we go, I've just changed one of them to this red, yellow, blue color scale. We've also got nice sequential color scales. So I'm going to change it here to the... I've forgotten the name of them.
09:01
I can never remember the names of these. That one, yep. We've also got the ability to change other symbology properties. So it's probably quite hard to actually see up there. So I'm going to go up, and I'm going to change the width of the lines, and so that will slowly update in real time. Whether this is a useful visualization or not is...
09:22
Yeah. It's very pretty, though. Yeah, I could look at this for a while. Anyway, yes, a tabular time series kind of work in the same way. You've got point sources, so you get that automatic styling. It knows it's a time series,
09:40
so it will bring up a little chart when you click on it, and you've got a little time slider at the bottom that will let you animate those color values. So this is coming from an Open DataSoft server in Australia. This is another time series chart of power generation in Australia. Another thing that you can do is combine these charts to do some sort of rough comparison.
10:04
So I've just selected two different power stations in Australia, and I'm just expanding the time series chart, and then you can see a nice kind of augmented chart at the bottom there that allows you to compare. We also get all the automatic styling stuff. This is a very silly example, as you'll see in a second,
10:20
but we've added the ability to change marker symbology. So here, I'm changing the point symbology by the fuel source. This video is a little bit slow, but yeah, changing by the fuel source, and as I go through the process, I should have timed this better. So I've got fossil fuel source there.
10:41
I'll give you a little water droplet. You know, it kind of makes sense. We've got solar there. We've got all the Mapbox maki icons in here, so you get a nice base of icons you can use. You can also copy and paste a URL to an image in there, and this is where it kind of gets silly because I've run out of ideas. So I said water mill, but obviously,
11:03
it shows you what you can do with this sort of stuff, and then it will update the legend so you can actually see what the data is showing. We also have region mapping, so this is your usual kind of, you know, chloropleth mapping, which joins tabular data with region geometry, vector tile-based.
11:21
We have a lot of different region sets, but unfortunately, they are mostly Australian-based, but we do provide a tool to let you generate your own. This is a nice region mapping example coming from the Australian Bureau of Statistics through their SDMX API, so we can pretty much bring up all of the public-facing statistical data
11:43
that's published by them. Time series region mapping automatically recognizes the time dimension. You can do the same thing. You can go and you can expand the charts. You can compare values over time for particular regions, and you can also edit the style as well if you want to. So analysis of open data.
12:02
Analysis is in quotes because it's very hand-wavy. It's the usual, I don't know, like it's not rigorous analysis or anything like that. It just provides a certain set of tools that you can use to do very qualitative, quick sort of analysis. So export there allows you to export data to do real analysis.
12:21
So we've got things like web coverage service export, which will let you take chunks of a WMS and get the raw raster data. We've also got vector tabular export as well. The compare tool I talked about lets you do that kind of screen space comparison between two different data sets. We've got a few other rudimentary tools
12:41
like measure, line of sight, which is in development, and a few more coming. We do have support for web processing service, so that's kind of as close as you get to real analysis. We also have a few specific open data cube analysis tools available. I can talk to you about that later
13:00
if you're interested. And we also have this new plugin support that we've developed, which allows you to extend that in a much easier way. So this is an example of the web processing service and our implementation. So this example calculates the mean value for monthly, vegetation, fractional cover, and precipitation.
13:22
So it accepts GeoJSON. Terrier talks to the WPS service. It recognizes this and it lets you draw a region on a map. It will also let you select existing regions, depending on which data sets you've got in your map. It's got a text field that lets you name the polygon. So it will then run this process
13:40
on that WPS server, service server, and it will just return a bunch of time series data sets you can then look at. So you can export these, you can overlay them, you can also share them if you want to. To the next and last part of exploring the open data, you can create and share maps. So everything that you do in Terrier.js
14:02
you can create unique share links that let you kind of capture the state of the map and share it with people. So the main way of doing this is we have a function, which I'm sure most of you understand. It creates a unique URL that captures the state of the map. So my lovely example
14:21
with custom marker symbology, you could share that with someone if you wanted to. We also have an embed function, which is very similar, but it just provides HTML to let you embed the map state into your own website. We also have a story function, which is similar to a share link, but it's got multiple scenes. Again,
14:42
another silly example that I've made up. So you can just create these scenes. You can give it a title and a tagline. So here I said, vegetation covering quite green here. And then you can go in and add more data, you can make any changes you want. So I go in and I add
15:01
the vegetation fractional cover, which is a little bit more interesting to look at. And then I zoom into Florence. But you can do anything here, you can create whatever kind of scenes you want with everything I've demonstrated so far. So I create another scene here, and I just notice that there's a little bit of red around Florence.
15:20
So I say there's a little bit of bare soil here. Very descriptive. I'm just going to skip forward here. So then once you've created this story, you can play it back and share it. So how is this useful to you? You don't live in Australia. What about your own open data? Well,
15:41
you can add your own data to any Terrier map, whether it's hosted by us or anyone. You can also drag and drop files in. And it's open source, you can host your own map if you want to. All of our open maps are the same Terrier code base with different configuration, just JSON configuration. So here's a nice example of an open data
16:01
portal that I've added to a map. So I'm really sorry about my Australian twang here. Bologna open data soft server. I've added this in. So they publish a lot of cool data for that province, or city I guess, but I'm not actually sure how wide the server goes. Anyway, so I'm typing
16:21
a URL to this open data soft server. This is all just happening client side. It adds the data. It's going to organize it through a few different things that open data soft provides. And so I'm going to look at theme. I've clicked on transport. I'm trying to find a layer of bus lines here, but I can't speak Italian. So there we go.
16:42
So yeah, it will show you all of the metadata that's available on that open data soft data set. It's going to have to change the map for you to Italy, because we're in Australia. And there you go. You've got a nice view of all of the locations. Also going to change the maps. You get a little bit more context, the base map. So you've got all the
17:01
automatic styling there. I change it to bus line. It's tried to visualize it as a kind of scalar quantity, which doesn't really make sense. So I change it to qualitative, and you can see all the bus lines now. So another example is drag and drop. So here I have
17:20
a 350 megabyte GeoJSON file that you can just drag onto the screen. This is of tree canopy data in one of the states in Australia. You've got a list of all the supporter files there on the left. So this video I've tried to keep in real time, because I didn't want to doctor it to show that it is there are delays as you do things, because there is just so much data.
17:43
So you click on a feature. It brings up all of the properties. I notice that there's this height property there. So I changed the styling to look at the height, because at the moment, tree area isn't really showing me very much. So it recomputes that styling. And as I zoom out, I notice that there's this really annoying white outline on everything.
18:02
So I go and change it. I'm just going to speed through this, because I'm running out of time. So I remove the border, and then I set the color scheme to something that's a little bit more pleasing. And then you get quite a nice visualization of the tree canopies in Melbourne.
18:23
So I also said you can create your own map. Terri.js is open source. You can do whatever you want with it. You know, obviously sticking to the license. MIT, Apache license. That's an example of a JSON config. Very simple. It's just describing an open data soft data set.
18:42
Just a single one in your catalog. Obviously, very customizable and extendable. You can push that as far as you want. Another thing to note is that if you do write your own Terrier JSON config, you actually don't need to host your own map. You can take that file, you can drag it onto an existing Terrier map and get the full functionality of
19:02
Terrier configuration. You can always extend Terrier. Open source project. Love finding new contributors. We have a new plugin system that makes it a lot easier to add your own tools. You can also develop support for other data portal services formats,
19:21
whatever you need. Not going to talk about that today, obviously. Future plans. So search data sets by location is asked for a lot, especially if you have 14,000 data sets. Searching via text doesn't really cut it. We're also looking at user workflows to add more complex data. I showed you that example
19:41
of adding that open data soft portal. Some data portals aren't going to let you just type in a URL and have a nice experience. You may do a few tweaks. So we're looking at adding UI workflows to do that. And thank you.
Recommendations
Series of 2 media