Logo TIB AV-Portal Logo TIB AV-Portal

Towards open, interoperable, and transdisciplinary point clouds for high performance computing

Video in TIB AV-Portal: Towards open, interoperable, and transdisciplinary point clouds for high performance computing

Formal Metadata

Towards open, interoperable, and transdisciplinary point clouds for high performance computing
Title of Series
Part Number
Number of Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
Large point clouds have emerged across a wide range of disciplines, however users and managers face a bewildering range of storage formats, large datasets and convoluted workflows for analysing point clouds alongside other data. Services like OpenTopography and the PDAL toolkit enable point cloud discovery and use, but integration with other earth systems data is not transparently supported. The Australian National Computational Infrastructure (NCI) hosts 10+PB of research data, predominantly in the realm of Earth Systems. These include extensive point cloud data which need to be discoverable alongside, and interoperable with, substantial collections of geospatial observations and model data using common tools in a High Performance Computing (HPC) and High Performance Data (HPD) environment. NCI has created a National Environmental Research Data Interoperability Platform (NERDIP) to help manage and analyse the data, both locally and remotely using web services which makes use of advanced features in HDF/NetCDF. We have demonstrated that deploying other geospatial data using a HDF5 model has the potential to directly improve large-scale usage and increase data interoperability between diverse geospatial collections. Models such as the Sensor Independent Point Cloud (SIPC) and SPDLib are based on HDF5. NCI are currently evaluating the use of these formats to aid discovery, extraction and processing using readily available tools, as well as interrogation via web services. The end goal for NCI is making point data discoverable and accessible to end-users in ways which allow seamless interoperability with other datasets and processing techniques.
Keywords Australian National Computational Infrastructure
Development PROM point part limitations open subsets Computational physics Computer animation PIE Meeting/Interview Rolling hill Gamma cloud
point services point verfolgen bits attitudes open subsets metadata mathematics Computational physics Computer animation different service organization pattern tasks cloud
point building time sources machine part virtual reality strategy operations case Em's square environment vertices input systems web Installer man portals interfaces point moment process HPC subset processes Computer animation service system cloud
point states point sources Contracts sets bits functions traces process Densities Densities Computer animation different orders Representation Dispersion manifolds environment Right waveform conditions
point formating time point sets process attributes CAN-bus static Computer animation Rolling environment sort metrics
user interfaces sources survey schemes shape infinity sun Emulation domain naturally level environment diagrams platforms area man sin formating model inner The list instance entire Computer animation case topology platforms website boil WAN
point words Computer animation Super point cloud Sum structure open subsets cloud data types
point current Slides current relation time point unit versions service Computer animation 5th sort tasks PDF
Computer animation directions 5th
point trajectories trajectories Computer animation files time point boundaries cloud bits sort
surface Slides Super model time mobile images sources horizon service different record aerodynamic design sources point moment model complex Databases part Densities Computer animation Symmetry case cloud life
sources Cut Computer animation Symmetry necessities pattern part sort Mass Sum
track beta files time formating process storage topology terms ascii text cloud standards varieties point survey moment metadata infinity capture Transformers data management Computer animation service topology different cloud testing sort current
point files load server point inner ease of use Databases Coloured number geometric Computer animation different Query testing testing sort cloud
point files waveform formating maximal lasers area training CNN Em's Indexable libraries cloud man point files Computing bits Actions non-existence Computer animation service testing sort pulse libraries source code
formating curve words geometric case Query Indexable spaces input mature cloud formating server point files The list part schemes Actions open subsets multithreaded words Computer animation cloud sort
point domain Cover building non-existence dots study closed moment polygons analysis experts water Computer animation terms different topology metrics
the so we're ready now for the last hour of the
session where finds dangerously move it away from the
development plants the more applied and part see also some of the limits of the tools and much of the thinking on yeah so you
don't have any magic amazingly technologies to show on the 1st day to have good story about an organization who's trying to manage a bunch of different as points and I will try to do and what we want to what we think we might be able to do next and so
he just a bit about where come from where a be computing node in Australia we store a lot of data pattern we have a big need to be know where that comes from and I what's happened to it and where it's going well what people are doing so the things like storing metadata with the data and provenance tracking a really be deals the so what we do
at the moment is mostly make a bunch of climate data and other systems that are available for people to computing jobs on and we have pretty much starting out on sitting data ViaWeb interfaces to people to build web portals on and and basically we just going to be growing job every time and these are the points we are going to be needing to so point cloud . and we're also going to be needing to the collects really be point clouds we don't have 500 billion point clouds 600 billion point clouds in here but I still use the 7 million square kilometers and it's eventually getting covered with with what our and other other sources of Don to we also as part of our
strategy we operate in OpenStack cloud interface solely on people can fire up a virtual machine and come to be computed on and do stuff so we installed we can install point cloud operational tools for people to work on and the machine without having to draw dotted anyway and so this is
just a little bit about what we do have and this data set here the camera is pretty much the only question right that in order to Australia and there's lots of lots of other light around but it's under a bunch of different licensing conditions console collected by different state agencies and this is kind of representative of what people are asking for contracts now so it's it's relatively dense it's full waveform dada and so it gets to reasonably be reasonably quickly
and here's another example different point out that we have so this is in what are the other role in the light of data small outputs these little points are aligned the four-dimensional traces for a lot of dispersal and we 42 about to be started no idea how many points we have that but I there are many billions and this is another source of point out that we we need to better manage in some way of so we're not talking just about like
and in a with that because it's pretty but we also you know the rest of Australia have geophysical data sets that are coming in as point out so far and and take magnetics already metrics or some other stuff and he comes to us and 16 in sort of some XYZ some other bunch of attributes formats and with the light to be arbitrary all of this started in roughly the same way because of it gets used by scientists and I come from a science background and I don't have time to win 20 different tools to manage data with 1 of the other get data and use of relatively easily so the
National Computational Infrastructure as a static interoperability approach which I sort of build building a
data infrastructure platform for from all the data that we collect and this this is this is a scheme that's kind of the intonation don't look at it from that In my obstructed talked about it's just 5 because this is like the basic level of everything and I is 1 way of doing things but I'm on the other side of the coin we so I will have to be moved out of different formats unless we need to and all those people this probably no real need to we can do other things that come but
in a small natural that the entire be diagram boils down to this 1 sentences in that we have of a lot of assistance data and we want to all work relatively well together so for instance as a scientist or a data user I want the other say I have geomagnetic surveys somewhere I wanna get the latest elevation model from the latest source also that area and look at magnetics and how they relate might be something as simple as this have no shape files embedded nesting sites enormously helpful trees are just things like that there are currently not all that easy said although we have some being here week now and my we've seen a lot of mind-blowing stuff that's gonna make list of holidays so on been a
fantastic learning journey sorry on this is my compulsory australian animals like that speed words some in trouble interoperable and transdisciplinary sorry we have a bunch of animals there and they're all relatively open moralistic enjoy and don't want you the kind of lessons but not there's definitely multiple disciplines there baby you can't do anything with and by way of merging and like it they're not they not interoperable and big so we have to get to that later mn and on
we also collect point cloud data like this this is sum the structures that make kids made I made this point that the mobile fine and this is a bunch of data we're also getting to store and manage and it's open a have become really do anything with it unless you consider during parenting of transdisciplinary
task on 0 this is meant to have a pretty picture that shows books time so if you get the PDF version of the slides that shows some of the traits that that we had from working nicely with so much in current that we also have 1 pretty much the same service and we basically pulling a points thing where they go in relation to ocean currents and I'm sorry that the picture doesn't play but there is a PDF version of this and when you get a little there the thank
is such sliding sorry this is a little and animation and I think we made this all my teammates made this in about have 14 units and it's pretty cool and that's the kind of stuff that we want to be able to do all the point out that we all the again is another
Sterling animal examples of here we have In some things that some things that don't necessarily look like they can work together but if you're request the 1 that has a direction Australia and you get it
gives us all if I not the way you expected to get some sort of
society by way of exploration sort of looked at ways to put points together In New Age just files and this is an example of some of my work we wanna keep complex ways that we wanna keep on uncertainty boundary uncertainty for all the points in the dataset and we want to keep things like aircraft trajectory so that if I want regenerate those points again if I go and look at them and go well you Thompson neural messed up my computational acting justly made them chemicals would other
data necessary just coincidence photogrammetric . crowd cloud it's stored in the same age defined we can just started in there and this is it takes a little bit of time to build a file that and when you want to play with that it's really easy to just get all that tolerance and put it together on OK so that the
messy slides here but so is it is another use case on the left hand side of go on let's talk and then we have a bunch of other stuff that we generate from the latter and this is important stuff that we wanna keep in the same place as well and then we have the uncertainties associated with those different services send so at the moment we well as a scientific user of point clouds it's it's difficult to do all this stuff in the same way but using what we know
the whole without going to move it to problem put them in a database also might still follow what we basically can't do with allows for or it impractical to use text or was things like that is a more complex example of we need to make this elevation model of Australia comes from a bunch of different sources of mystery is the around pretty quickly it's the fastest moving continent on the planet and we want be and take all their life and all that the symmetry data that we've got a source the symmetry we wanna take on any of basically all of the elevation data we can get hold of nouns national together relatively frequently is currently takes a long time to build a model and you kind of trying to find a way to do it better sound this is another 1
of the problems and all Australia data pattern so we got some minor cuts and necessity and some sort of symmetry here and in this massive debt nothing say on we want be out instead of going well we need to manage this stuff over here
1 way and then all this stuff you away and stuff in other ways and it makes it really hard to then go will we kind of need to do things freeways and then managed to figure out what's in here as well harming yeah this is where
we're at at the moment point cloud data in Australia still considered difficult stuff but there's a lot of innovative things going on in terms of collecting and doing small deals that but in terms of the beta tree nodes it's it's kind of saying it is hard to manage thing because it's not a regular grid of you know files as a big and it tends to just get turned into our cells and put away and that's that's is somewhat lower cost at once every now and then but because the data are hard to use and nobody really tries to do anything with them in and here
I can some ways to put down point clouds into files that we know that but we have some as a mentioned at the start of the talk of got some specific needs in that some whatever we do with the files we need to be at track what happens to them and figure out of the process recreate infinity all around here basically know where they come from and where they're going from and we like to maintain some sort of standard compliance and and keep things fast and easy to access food and columns talking incredibly useful to us and so this is some ways that we ignore at a certain point of and these basically experiments that we need to do yet I sort of putting my abstract away from 6 months to make something cool and got waylaid along the way so time just getting started on who have done a few
things so 1 of our test here I'm always been working the with price this point and number so far far I'm quite happy with sort of come from a background of working with laser laser scanner pointing files and then the other sort of stuff to do and cost of doing pointing to database in you know and then pull back out really quickly I just made this I present book he it
takes some 45 minutes to query the whole less than 5 minutes it's it's really quick to build this picture of newly billion points and on on the left there is also of a mountain retirement and the camera and on here is 3 different classifications color separately and that's just really nice quick a going towards
some yes some things we deny before I came here very young work with this so far has led us to think will the that this is still quite a bit of can be added to make them smaller which is great and I can just keep going and figuring out what we need to do on just quickly we did
try some stiff based stuff and a natural anyone knows about the sort of of people start a library but it's a it's a way of storing points in a hasty file that's indexed by a laser pulse and looks quite nice but it's we can get to work and we need to teaching training and probably helped politics small work come in and I'm at the
end of the day we recognize that whatever we choose to do there are lots of existing workflows that use formats widely accepted and we don't break the and we don't want necessarily blow away all of our existing data that exists we need to just work in a way to keep and now this is basically just a word cloud of stuff we've been thinking about but on the outskirts of mature and so down a what enough you know I mean a lot of the stuff we've seen today and this week has been really useful for sort of guiding where going with all of this stuff but and here is a list of things to community that we highly depend on and on the yeah is cold really useful and and gray and that's the end thank
you thank you and any reaction questions yes he and when when
you say we have a we have lots of different answer which we actually want to match to the that that matches what you and what what what what did you have in mind is that like a your you want to do and the big dots analysis where you're never sure what's going to come out um or you only have in mind what you wanted to to mention to each other on so that in terms of building like national elevation molar intensive some scientists coming on and writing down the engines also because we we basically so that and we have no idea what users are going to do so in a way where would kind of mandated to make access to the data as easy as possible so and we want to enable so those things that haven't been easy to do in the past like there is I think in existence of study where some scientists in Australia validated spot 5 tree cover metrics using a 1 liner and was a big study and it was it was not so easy for them to do and in a way we would like to be out get people to reproduce that just on anywhere in the country just grab a chunk of data they began a polygon and I will get the the spot immediately the ladder and was do stuff with and without having to become domain experts and how to access and analyze water or other point on other questions yeah look at moment and thanks to anonymous and this closes the session of right now and then we have the closing plenary keynote and closing session and thank you