Towards open, interoperable, and transdisciplinary point clouds for high performance computing

Video in TIB AV-Portal: Towards open, interoperable, and transdisciplinary point clouds for high performance computing

13 views

Formal Metadata

Title
Towards open, interoperable, and transdisciplinary point clouds for high performance computing
Title of Series
Part Number
162
Number of Parts
193
Author
Steer, Adam
License
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
FOSS4G, Open Source Geospatial Foundation (OSGeo)
Release Date
2016
Language
English

Content Metadata

Subject Area
Abstract
Large point clouds have emerged across a wide range of disciplines, however users and managers face a bewildering range of storage formats, large datasets and convoluted workflows for analysing point clouds alongside other data. Services like OpenTopography and the PDAL toolkit enable point cloud discovery and use, but integration with other earth systems data is not transparently supported. The Australian National Computational Infrastructure (NCI) hosts 10+PB of research data, predominantly in the realm of Earth Systems. These include extensive point cloud data which need to be discoverable alongside, and interoperable with, substantial collections of geospatial observations and model data using common tools in a High Performance Computing (HPC) and High Performance Data (HPD) environment. NCI has created a National Environmental Research Data Interoperability Platform (NERDIP) to help manage and analyse the data, both locally and remotely using web services which makes use of advanced features in HDF/NetCDF. We have demonstrated that deploying other geospatial data using a HDF5 model has the potential to directly improve large-scale usage and increase data interoperability between diverse geospatial collections. Models such as the Sensor Independent Point Cloud (SIPC) and SPDLib are based on HDF5. NCI are currently evaluating the use of these formats to aid discovery, extraction and processing using readily available tools, as well as interrogation via web services. The end goal for NCI is making point data discoverable and accessible to end-users in ways which allow seamless interoperability with other datasets and processing techniques.
Keywords
Australian National Computational Infrastructure
Loading...
Software developer Programmable read-only memory Point (geometry) Point cloud Limit (category theory) Mereology Open set Computer animation Pi Meeting/Interview Computational science Moving average Hill differential equation Gamma function
Point (geometry) Service (economics) Point (geometry) Video tracking Point cloud Bit Euler angles Open set Metadata Computer animation Mathematics Web service Vertex (graph theory) Computational science Self-organization Pattern language Subtraction Task (computing)
Point (geometry) Building Multiplication sign Source code Process modeling Virtual machine Point cloud Mereology Virtual reality Strategy game Operator (mathematics) Personal digital assistant Electronic meeting system Square number Integrated development environment Vertex (graph theory) output Physical system World Wide Web Consortium Installation art Metropolitan area network Web portal Point (geometry) Moment (mathematics) Interface (computing) Supercomputer Subset Process (computing) Computer animation Web service System programming
Point (geometry) State of matter Point (geometry) Source code Process modeling Design by contract Bit Manifold Set (mathematics) Function (mathematics) Tracing (software) Population density Population density Computer animation Order (biology) Representation (politics) Dispersion (chemistry) Integrated development environment Waveform Right angle Subtraction Condition number
Point (geometry) File format Multiplication sign Point (geometry) Process modeling Set (mathematics) Metric tensor Attribute grammar Fluid statics Computer animation Moving average Integrated development environment Quicksort
Numbering scheme User interface Scientific modelling Source code Archaeological field survey Shape (magazine) Infinity Emulation Natural number Energy level Integrated development environment Diagram Computing platform Area Metropolitan area network Sine File format Interior (topology) Electronic mailing list Instance (computer science) Entire function Computer animation Network topology Personal digital assistant Computing platform Website Boiling point Domain name Wide area network
Point (geometry) Word Summation Computer animation Inheritance (object-oriented programming) Point (geometry) Point cloud Point cloud Data structure Open set Data type
Point (geometry) Ocean current Slide rule Ocean current Theory of relativity Multiplication sign Point (geometry) Revision control Web service Computer animation 5 (number) Quicksort Units of measurement Task (computing) Probability density function
Computer animation Direction (geometry) 5 (number)
Point (geometry) Computer animation Computer file Multiplication sign Point (geometry) Trajectory Boundary value problem Bit Point cloud Quicksort Trajectory
Surface Slide rule Inheritance (object-oriented programming) Multiplication sign Scientific modelling Mobile Web Computer-generated imagery Source code Point cloud Horizon Data model Web service Video game Database Row (database) Aerodynamics Subtraction Source code Point (geometry) Moment (mathematics) Complex (psychology) Mereology Population density Computer animation Symmetry (physics) Personal digital assistant
Source code Summation Computer animation Symmetry (physics) Modal logic Pattern language Mereology Quicksort Mass Film editing
Trail Beta function Computer file Multiplication sign File format Point cloud Term (mathematics) Network topology Data storage device ASCII Standard deviation Variety (linguistics) Process modeling Point (geometry) Archaeological field survey Moment (mathematics) Metadata Infinity Point cloud Motion capture Transformation (genetics) Computer animation Network topology Web service Vertex (graph theory) Software testing Quicksort Data management Electric current
Point (geometry) Query language Computer file Structural load Geometry Server (computing) Point (geometry) Interior (topology) Point cloud Usability Number Computer animation Graph coloring Database Software testing Software testing Quicksort Subtraction
Point (geometry) Computer file Waveform File format Laser Area Wave packet CNN Electronic meeting system Subject indexing Library (computing) Metropolitan area network Point (geometry) Computer file Coma Berenices Point cloud Bit Group theory Existence Maxima and minima Computer animation Web service Data compression Software testing Quicksort Pulse (signal processing) Library (computing)
Query language Spacetime Numbering scheme File format Geometry Server (computing) Point (geometry) Computer file File format Curve Electronic mailing list Point cloud Point cloud Mereology Group theory Open set Thread (computing) Word Word Computer animation Subject indexing Quicksort output Capability Maturity Model
Covering space Point (geometry) Domain name Dot product Building Existence Observational study Closed set Polygon Moment (mathematics) Expert system Mathematical analysis Water vapor Metric tensor Computer animation Network topology Term (mathematics) Subtraction
the so we're ready now for the last hour of the
session where finds dangerously move it away from the
development plants the more applied and part see also some of the limits of the tools and much of the thinking on yeah so you
don't have any magic amazingly technologies to show on the 1st day to have good story about an organization who's trying to manage a bunch of different as points and I will try to do and what we want to what we think we might be able to do next and so
he just a bit about where come from where a be computing node in Australia we store a lot of data pattern we have a big need to be know where that comes from and I what's happened to it and where it's going well what people are doing so the things like storing metadata with the data and provenance tracking a really be deals the so what we do
at the moment is mostly make a bunch of climate data and other systems that are available for people to computing jobs on and we have pretty much starting out on sitting data ViaWeb interfaces to people to build web portals on and and basically we just going to be growing job every time and these are the points we are going to be needing to so point cloud . and we're also going to be needing to the collects really be point clouds we don't have 500 billion point clouds 600 billion point clouds in here but I still use the 7 million square kilometers and it's eventually getting covered with with what our and other other sources of Don to we also as part of our
strategy we operate in OpenStack cloud interface solely on people can fire up a virtual machine and come to be computed on and do stuff so we installed we can install point cloud operational tools for people to work on and the machine without having to draw dotted anyway and so this is
just a little bit about what we do have and this data set here the camera is pretty much the only question right that in order to Australia and there's lots of lots of other light around but it's under a bunch of different licensing conditions console collected by different state agencies and this is kind of representative of what people are asking for contracts now so it's it's relatively dense it's full waveform dada and so it gets to reasonably be reasonably quickly
and here's another example different point out that we have so this is in what are the other role in the light of data small outputs these little points are aligned the four-dimensional traces for a lot of dispersal and we 42 about to be started no idea how many points we have that but I there are many billions and this is another source of point out that we we need to better manage in some way of so we're not talking just about like
and in a with that because it's pretty but we also you know the rest of Australia have geophysical data sets that are coming in as point out so far and and take magnetics already metrics or some other stuff and he comes to us and 16 in sort of some XYZ some other bunch of attributes formats and with the light to be arbitrary all of this started in roughly the same way because of it gets used by scientists and I come from a science background and I don't have time to win 20 different tools to manage data with 1 of the other get data and use of relatively easily so the
National Computational Infrastructure as a static interoperability approach which I sort of build building a
data infrastructure platform for from all the data that we collect and this this is this is a scheme that's kind of the intonation don't look at it from that In my obstructed talked about it's just 5 because this is like the basic level of everything and I is 1 way of doing things but I'm on the other side of the coin we so I will have to be moved out of different formats unless we need to and all those people this probably no real need to we can do other things that come but
in a small natural that the entire be diagram boils down to this 1 sentences in that we have of a lot of assistance data and we want to all work relatively well together so for instance as a scientist or a data user I want the other say I have geomagnetic surveys somewhere I wanna get the latest elevation model from the latest source also that area and look at magnetics and how they relate might be something as simple as this have no shape files embedded nesting sites enormously helpful trees are just things like that there are currently not all that easy said although we have some being here week now and my we've seen a lot of mind-blowing stuff that's gonna make list of holidays so on been a
fantastic learning journey sorry on this is my compulsory australian animals like that speed words some in trouble interoperable and transdisciplinary sorry we have a bunch of animals there and they're all relatively open moralistic enjoy and don't want you the kind of lessons but not there's definitely multiple disciplines there baby you can't do anything with and by way of merging and like it they're not they not interoperable and big so we have to get to that later mn and on
we also collect point cloud data like this this is sum the structures that make kids made I made this point that the mobile fine and this is a bunch of data we're also getting to store and manage and it's open a have become really do anything with it unless you consider during parenting of transdisciplinary
task on 0 this is meant to have a pretty picture that shows books time so if you get the PDF version of the slides that shows some of the traits that that we had from working nicely with so much in current that we also have 1 pretty much the same service and we basically pulling a points thing where they go in relation to ocean currents and I'm sorry that the picture doesn't play but there is a PDF version of this and when you get a little there the thank
is such sliding sorry this is a little and animation and I think we made this all my teammates made this in about have 14 units and it's pretty cool and that's the kind of stuff that we want to be able to do all the point out that we all the again is another
Sterling animal examples of here we have In some things that some things that don't necessarily look like they can work together but if you're request the 1 that has a direction Australia and you get it
gives us all if I not the way you expected to get some sort of
society by way of exploration sort of looked at ways to put points together In New Age just files and this is an example of some of my work we wanna keep complex ways that we wanna keep on uncertainty boundary uncertainty for all the points in the dataset and we want to keep things like aircraft trajectory so that if I want regenerate those points again if I go and look at them and go well you Thompson neural messed up my computational acting justly made them chemicals would other
data necessary just coincidence photogrammetric . crowd cloud it's stored in the same age defined we can just started in there and this is it takes a little bit of time to build a file that and when you want to play with that it's really easy to just get all that tolerance and put it together on OK so that the
messy slides here but so is it is another use case on the left hand side of go on let's talk and then we have a bunch of other stuff that we generate from the latter and this is important stuff that we wanna keep in the same place as well and then we have the uncertainties associated with those different services send so at the moment we well as a scientific user of point clouds it's it's difficult to do all this stuff in the same way but using what we know
the whole without going to move it to problem put them in a database also might still follow what we basically can't do with allows for or it impractical to use text or was things like that is a more complex example of we need to make this elevation model of Australia comes from a bunch of different sources of mystery is the around pretty quickly it's the fastest moving continent on the planet and we want be and take all their life and all that the symmetry data that we've got a source the symmetry we wanna take on any of basically all of the elevation data we can get hold of nouns national together relatively frequently is currently takes a long time to build a model and you kind of trying to find a way to do it better sound this is another 1
of the problems and all Australia data pattern so we got some minor cuts and necessity and some sort of symmetry here and in this massive debt nothing say on we want be out instead of going well we need to manage this stuff over here
1 way and then all this stuff you away and stuff in other ways and it makes it really hard to then go will we kind of need to do things freeways and then managed to figure out what's in here as well harming yeah this is where
we're at at the moment point cloud data in Australia still considered difficult stuff but there's a lot of innovative things going on in terms of collecting and doing small deals that but in terms of the beta tree nodes it's it's kind of saying it is hard to manage thing because it's not a regular grid of you know files as a big and it tends to just get turned into our cells and put away and that's that's is somewhat lower cost at once every now and then but because the data are hard to use and nobody really tries to do anything with them in and here
I can some ways to put down point clouds into files that we know that but we have some as a mentioned at the start of the talk of got some specific needs in that some whatever we do with the files we need to be at track what happens to them and figure out of the process recreate infinity all around here basically know where they come from and where they're going from and we like to maintain some sort of standard compliance and and keep things fast and easy to access food and columns talking incredibly useful to us and so this is some ways that we ignore at a certain point of and these basically experiments that we need to do yet I sort of putting my abstract away from 6 months to make something cool and got waylaid along the way so time just getting started on who have done a few
things so 1 of our test here I'm always been working the with price this point and number so far far I'm quite happy with sort of come from a background of working with laser laser scanner pointing files and then the other sort of stuff to do and cost of doing pointing to database in you know and then pull back out really quickly I just made this I present book he it
takes some 45 minutes to query the whole less than 5 minutes it's it's really quick to build this picture of newly billion points and on on the left there is also of a mountain retirement and the camera and on here is 3 different classifications color separately and that's just really nice quick a going towards
some yes some things we deny before I came here very young work with this so far has led us to think will the that this is still quite a bit of can be added to make them smaller which is great and I can just keep going and figuring out what we need to do on just quickly we did
try some stiff based stuff and a natural anyone knows about the sort of of people start a library but it's a it's a way of storing points in a hasty file that's indexed by a laser pulse and looks quite nice but it's we can get to work and we need to teaching training and probably helped politics small work come in and I'm at the
end of the day we recognize that whatever we choose to do there are lots of existing workflows that use formats widely accepted and we don't break the and we don't want necessarily blow away all of our existing data that exists we need to just work in a way to keep and now this is basically just a word cloud of stuff we've been thinking about but on the outskirts of mature and so down a what enough you know I mean a lot of the stuff we've seen today and this week has been really useful for sort of guiding where going with all of this stuff but and here is a list of things to community that we highly depend on and on the yeah is cold really useful and and gray and that's the end thank
you thank you and any reaction questions yes he and when when
you say we have a we have lots of different answer which we actually want to match to the that that matches what you and what what what what did you have in mind is that like a your you want to do and the big dots analysis where you're never sure what's going to come out um or you only have in mind what you wanted to to mention to each other on so that in terms of building like national elevation molar intensive some scientists coming on and writing down the engines also because we we basically so that and we have no idea what users are going to do so in a way where would kind of mandated to make access to the data as easy as possible so and we want to enable so those things that haven't been easy to do in the past like there is I think in existence of study where some scientists in Australia validated spot 5 tree cover metrics using a 1 liner and was a big study and it was it was not so easy for them to do and in a way we would like to be out get people to reproduce that just on anywhere in the country just grab a chunk of data they began a polygon and I will get the the spot immediately the ladder and was do stuff with and without having to become domain experts and how to access and analyze water or other point on other questions yeah look at moment and thanks to anonymous and this closes the session of right now and then we have the closing plenary keynote and closing session and thank you
Loading...
Feedback

Timings

  526 ms - page object

Version

AV-Portal 3.13.1 (abea844c86ad1b15ca76e1472346f3fd8bea123a)
hidden