Towards open, interoperable, and transdisciplinary point clouds for high performance computing

Video in TIB AV-Portal: Towards open, interoperable, and transdisciplinary point clouds for high performance computing

Formal Metadata

Towards open, interoperable, and transdisciplinary point clouds for high performance computing
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Large point clouds have emerged across a wide range of disciplines, however users and managers face a bewildering range of storage formats, large datasets and convoluted workflows for analysing point clouds alongside other data. Services like OpenTopography and the PDAL toolkit enable point cloud discovery and use, but integration with other earth systems data is not transparently supported. The Australian National Computational Infrastructure (NCI) hosts 10+PB of research data, predominantly in the realm of Earth Systems. These include extensive point cloud data which need to be discoverable alongside, and interoperable with, substantial collections of geospatial observations and model data using common tools in a High Performance Computing (HPC) and High Performance Data (HPD) environment. NCI has created a National Environmental Research Data Interoperability Platform (NERDIP) to help manage and analyse the data, both locally and remotely using web services which makes use of advanced features in HDF/NetCDF. We have demonstrated that deploying other geospatial data using a HDF5 model has the potential to directly improve large-scale usage and increase data interoperability between diverse geospatial collections. Models such as the Sensor Independent Point Cloud (SIPC) and SPDLib are based on HDF5. NCI are currently evaluating the use of these formats to aid discovery, extraction and processing using readily available tools, as well as interrogation via web services. The end goal for NCI is making point data discoverable and accessible to end-users in ways which allow seamless interoperability with other datasets and processing techniques.
Keywords Australian National Computational Infrastructure
Computational physics Pi Software developer Programmable read-only memory Point (geometry) Moving average Hill differential equation Gamma function Limit (category theory) Mereology Open set Point cloud
Point (geometry) Service (economics) Point (geometry) Video tracking Bit Euler angles Open set Metadata Mathematics Computational physics Different (Kate Ryan album) Web service Self-organization Pattern language Task (computing) Point cloud
Point (geometry) Building Multiplication sign Source code Virtual machine Mereology Virtual reality Strategy game Operator (mathematics) Personal digital assistant Electronic meeting system Square number Integrated development environment Vertex (graph theory) output Physical system World Wide Web Consortium Installation art Metropolitan area network Web portal Interface (computing) Point (geometry) Moment (mathematics) Process modeling Supercomputer Subset Process (computing) Web service System programming Point cloud
Point (geometry) State of matter Point (geometry) Source code Set (mathematics) Design by contract Bit Function (mathematics) Tracing (software) Process modeling Population density Population density Different (Kate Ryan album) Order (biology) Representation (politics) Dispersion (chemistry) Manifold Integrated development environment Right angle Waveform Condition number
Point (geometry) File format Multiplication sign Point (geometry) Set (mathematics) Process modeling Attribute grammar CAN bus Fluid statics Moving average Integrated development environment Quicksort Metric system
User interface Source code Archaeological field survey Numbering scheme Shape (magazine) Infinity Special unitary group Emulation Time domain Natural number Energy level Integrated development environment Diagram Endliche Modelltheorie Computing platform Area Metropolitan area network Sine File format Interior (topology) Electronic mailing list Instance (computer science) Entire function Personal digital assistant Network topology Computing platform Website Boiling point Wide area network
Point (geometry) Word Inheritance (object-oriented programming) Point (geometry) Point cloud Summierbarkeit Data structure Open set Point cloud Data type
Ocean current Point (geometry) Slide rule Ocean current Theory of relativity Multiplication sign Point (geometry) Execution unit Revision control Web service 5 (number) Quicksort Task (computing) Probability density function
Direction (geometry) 5 (number)
Point (geometry) Trajectory Trajectory Computer file Multiplication sign Point (geometry) Boundary value problem Point cloud Bit Quicksort
Surface Slide rule Inheritance (object-oriented programming) Multiplication sign Mobile Web Computer-generated imagery Source code Horizon Data model Web service Different (Kate Ryan album) Row (database) Aerodynamics Endliche Modelltheorie Source code Point (geometry) Moment (mathematics) Complex (psychology) Database Mereology Population density Symmetry (physics) Personal digital assistant Point cloud Video game
Source code Film editing Symmetry (physics) Modal logic Pattern language Mereology Mass Quicksort Summierbarkeit
Trail Beta function Computer file Multiplication sign File format Process modeling Data storage device Network topology Term (mathematics) ASCII Point cloud Standard deviation Variety (linguistics) Point (geometry) Archaeological field survey Moment (mathematics) Metadata Infinity Motion capture Transformation (genetics) Data management Web service Network topology Different (Kate Ryan album) Point cloud Software testing Quicksort Electric current
Point (geometry) Computer file Structural load Server (computing) Point (geometry) Interior (topology) Usability Database Graph coloring Number Geometry Different (Kate Ryan album) Query language Software testing Software testing Quicksort Point cloud
Point (geometry) Computer file Waveform File format Maxima and minima Laser Area Wave packet CNN Electronic meeting system Subject indexing Library (computing) Point cloud Metropolitan area network Point (geometry) Computer file Coma Berenices Bit Group action Existence Web service Software testing Quicksort Pulse (signal processing) Library (computing) Data compression
File format Server (computing) Point (geometry) Computer file File format Curve Electronic mailing list Mereology Numbering scheme Group action Open set Thread (computing) Word Geometry Word Personal digital assistant Query language Subject indexing Point cloud Spacetime Quicksort output Capability Maturity Model Point cloud
Point (geometry) Domain name Covering space Building Existence Dot product Observational study Closed set Moment (mathematics) Polygon Mathematical analysis Expert system Water vapor Term (mathematics) Different (Kate Ryan album) Network topology Metric system
the so we're ready now for the last hour of the
session where finds dangerously move it away from the
development plants the more applied and part see also some of the limits of the tools and much of the thinking on yeah so you
don't have any magic amazingly technologies to show on the 1st day to have good story about an organization who's trying to manage a bunch of different as points and I will try to do and what we want to what we think we might be able to do next and so
he just a bit about where come from where a be computing node in Australia we store a lot of data pattern we have a big need to be know where that comes from and I what's happened to it and where it's going well what people are doing so the things like storing metadata with the data and provenance tracking a really be deals the so what we do
at the moment is mostly make a bunch of climate data and other systems that are available for people to computing jobs on and we have pretty much starting out on sitting data ViaWeb interfaces to people to build web portals on and and basically we just going to be growing job every time and these are the points we are going to be needing to so point cloud . and we're also going to be needing to the collects really be point clouds we don't have 500 billion point clouds 600 billion point clouds in here but I still use the 7 million square kilometers and it's eventually getting covered with with what our and other other sources of Don to we also as part of our
strategy we operate in OpenStack cloud interface solely on people can fire up a virtual machine and come to be computed on and do stuff so we installed we can install point cloud operational tools for people to work on and the machine without having to draw dotted anyway and so this is
just a little bit about what we do have and this data set here the camera is pretty much the only question right that in order to Australia and there's lots of lots of other light around but it's under a bunch of different licensing conditions console collected by different state agencies and this is kind of representative of what people are asking for contracts now so it's it's relatively dense it's full waveform dada and so it gets to reasonably be reasonably quickly
and here's another example different point out that we have so this is in what are the other role in the light of data small outputs these little points are aligned the four-dimensional traces for a lot of dispersal and we 42 about to be started no idea how many points we have that but I there are many billions and this is another source of point out that we we need to better manage in some way of so we're not talking just about like
and in a with that because it's pretty but we also you know the rest of Australia have geophysical data sets that are coming in as point out so far and and take magnetics already metrics or some other stuff and he comes to us and 16 in sort of some XYZ some other bunch of attributes formats and with the light to be arbitrary all of this started in roughly the same way because of it gets used by scientists and I come from a science background and I don't have time to win 20 different tools to manage data with 1 of the other get data and use of relatively easily so the
National Computational Infrastructure as a static interoperability approach which I sort of build building a
data infrastructure platform for from all the data that we collect and this this is this is a scheme that's kind of the intonation don't look at it from that In my obstructed talked about it's just 5 because this is like the basic level of everything and I is 1 way of doing things but I'm on the other side of the coin we so I will have to be moved out of different formats unless we need to and all those people this probably no real need to we can do other things that come but
in a small natural that the entire be diagram boils down to this 1 sentences in that we have of a lot of assistance data and we want to all work relatively well together so for instance as a scientist or a data user I want the other say I have geomagnetic surveys somewhere I wanna get the latest elevation model from the latest source also that area and look at magnetics and how they relate might be something as simple as this have no shape files embedded nesting sites enormously helpful trees are just things like that there are currently not all that easy said although we have some being here week now and my we've seen a lot of mind-blowing stuff that's gonna make list of holidays so on been a
fantastic learning journey sorry on this is my compulsory australian animals like that speed words some in trouble interoperable and transdisciplinary sorry we have a bunch of animals there and they're all relatively open moralistic enjoy and don't want you the kind of lessons but not there's definitely multiple disciplines there baby you can't do anything with and by way of merging and like it they're not they not interoperable and big so we have to get to that later mn and on
we also collect point cloud data like this this is sum the structures that make kids made I made this point that the mobile fine and this is a bunch of data we're also getting to store and manage and it's open a have become really do anything with it unless you consider during parenting of transdisciplinary
task on 0 this is meant to have a pretty picture that shows books time so if you get the PDF version of the slides that shows some of the traits that that we had from working nicely with so much in current that we also have 1 pretty much the same service and we basically pulling a points thing where they go in relation to ocean currents and I'm sorry that the picture doesn't play but there is a PDF version of this and when you get a little there the thank
is such sliding sorry this is a little and animation and I think we made this all my teammates made this in about have 14 units and it's pretty cool and that's the kind of stuff that we want to be able to do all the point out that we all the again is another
Sterling animal examples of here we have In some things that some things that don't necessarily look like they can work together but if you're request the 1 that has a direction Australia and you get it
gives us all if I not the way you expected to get some sort of
society by way of exploration sort of looked at ways to put points together In New Age just files and this is an example of some of my work we wanna keep complex ways that we wanna keep on uncertainty boundary uncertainty for all the points in the dataset and we want to keep things like aircraft trajectory so that if I want regenerate those points again if I go and look at them and go well you Thompson neural messed up my computational acting justly made them chemicals would other
data necessary just coincidence photogrammetric . crowd cloud it's stored in the same age defined we can just started in there and this is it takes a little bit of time to build a file that and when you want to play with that it's really easy to just get all that tolerance and put it together on OK so that the
messy slides here but so is it is another use case on the left hand side of go on let's talk and then we have a bunch of other stuff that we generate from the latter and this is important stuff that we wanna keep in the same place as well and then we have the uncertainties associated with those different services send so at the moment we well as a scientific user of point clouds it's it's difficult to do all this stuff in the same way but using what we know
the whole without going to move it to problem put them in a database also might still follow what we basically can't do with allows for or it impractical to use text or was things like that is a more complex example of we need to make this elevation model of Australia comes from a bunch of different sources of mystery is the around pretty quickly it's the fastest moving continent on the planet and we want be and take all their life and all that the symmetry data that we've got a source the symmetry we wanna take on any of basically all of the elevation data we can get hold of nouns national together relatively frequently is currently takes a long time to build a model and you kind of trying to find a way to do it better sound this is another 1
of the problems and all Australia data pattern so we got some minor cuts and necessity and some sort of symmetry here and in this massive debt nothing say on we want be out instead of going well we need to manage this stuff over here
1 way and then all this stuff you away and stuff in other ways and it makes it really hard to then go will we kind of need to do things freeways and then managed to figure out what's in here as well harming yeah this is where
we're at at the moment point cloud data in Australia still considered difficult stuff but there's a lot of innovative things going on in terms of collecting and doing small deals that but in terms of the beta tree nodes it's it's kind of saying it is hard to manage thing because it's not a regular grid of you know files as a big and it tends to just get turned into our cells and put away and that's that's is somewhat lower cost at once every now and then but because the data are hard to use and nobody really tries to do anything with them in and here
I can some ways to put down point clouds into files that we know that but we have some as a mentioned at the start of the talk of got some specific needs in that some whatever we do with the files we need to be at track what happens to them and figure out of the process recreate infinity all around here basically know where they come from and where they're going from and we like to maintain some sort of standard compliance and and keep things fast and easy to access food and columns talking incredibly useful to us and so this is some ways that we ignore at a certain point of and these basically experiments that we need to do yet I sort of putting my abstract away from 6 months to make something cool and got waylaid along the way so time just getting started on who have done a few
things so 1 of our test here I'm always been working the with price this point and number so far far I'm quite happy with sort of come from a background of working with laser laser scanner pointing files and then the other sort of stuff to do and cost of doing pointing to database in you know and then pull back out really quickly I just made this I present book he it
takes some 45 minutes to query the whole less than 5 minutes it's it's really quick to build this picture of newly billion points and on on the left there is also of a mountain retirement and the camera and on here is 3 different classifications color separately and that's just really nice quick a going towards
some yes some things we deny before I came here very young work with this so far has led us to think will the that this is still quite a bit of can be added to make them smaller which is great and I can just keep going and figuring out what we need to do on just quickly we did
try some stiff based stuff and a natural anyone knows about the sort of of people start a library but it's a it's a way of storing points in a hasty file that's indexed by a laser pulse and looks quite nice but it's we can get to work and we need to teaching training and probably helped politics small work come in and I'm at the
end of the day we recognize that whatever we choose to do there are lots of existing workflows that use formats widely accepted and we don't break the and we don't want necessarily blow away all of our existing data that exists we need to just work in a way to keep and now this is basically just a word cloud of stuff we've been thinking about but on the outskirts of mature and so down a what enough you know I mean a lot of the stuff we've seen today and this week has been really useful for sort of guiding where going with all of this stuff but and here is a list of things to community that we highly depend on and on the yeah is cold really useful and and gray and that's the end thank
you thank you and any reaction questions yes he and when when
you say we have a we have lots of different answer which we actually want to match to the that that matches what you and what what what what did you have in mind is that like a your you want to do and the big dots analysis where you're never sure what's going to come out um or you only have in mind what you wanted to to mention to each other on so that in terms of building like national elevation molar intensive some scientists coming on and writing down the engines also because we we basically so that and we have no idea what users are going to do so in a way where would kind of mandated to make access to the data as easy as possible so and we want to enable so those things that haven't been easy to do in the past like there is I think in existence of study where some scientists in Australia validated spot 5 tree cover metrics using a 1 liner and was a big study and it was it was not so easy for them to do and in a way we would like to be out get people to reproduce that just on anywhere in the country just grab a chunk of data they began a polygon and I will get the the spot immediately the ladder and was do stuff with and without having to become domain experts and how to access and analyze water or other point on other questions yeah look at moment and thanks to anonymous and this closes the session of right now and then we have the closing plenary keynote and closing session and thank you


  154 ms - page object


AV-Portal 3.20.2 (36f6df173ce4850b467c9cb7af359cf1cdaed247)