Add to Watchlist

Towards open, interoperable, and transdisciplinary point clouds for high performance computing

10 views

Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Towards open, interoperable, and transdisciplinary point clouds for high performance computing
Title of Series FOSS4G Bonn 2016
Part Number 162
Number of Parts 193
Author Steer, Adam
License CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
DOI 10.5446/20440
Publisher FOSS4G, Open Source Geospatial Foundation (OSGeo)
Release Date 2016
Language English

Content Metadata

Subject Area Computer Science
Abstract Large point clouds have emerged across a wide range of disciplines, however users and managers face a bewildering range of storage formats, large datasets and convoluted workflows for analysing point clouds alongside other data. Services like OpenTopography and the PDAL toolkit enable point cloud discovery and use, but integration with other earth systems data is not transparently supported. The Australian National Computational Infrastructure (NCI) hosts 10+PB of research data, predominantly in the realm of Earth Systems. These include extensive point cloud data which need to be discoverable alongside, and interoperable with, substantial collections of geospatial observations and model data using common tools in a High Performance Computing (HPC) and High Performance Data (HPD) environment. NCI has created a National Environmental Research Data Interoperability Platform (NERDIP) to help manage and analyse the data, both locally and remotely using web services which makes use of advanced features in HDF/NetCDF. We have demonstrated that deploying other geospatial data using a HDF5 model has the potential to directly improve large-scale usage and increase data interoperability between diverse geospatial collections. Models such as the Sensor Independent Point Cloud (SIPC) and SPDLib are based on HDF5. NCI are currently evaluating the use of these formats to aid discovery, extraction and processing using readily available tools, as well as interrogation via web services. The end goal for NCI is making point data discoverable and accessible to end-users in ways which allow seamless interoperability with other datasets and processing techniques.
Keywords Australian National Computational Infrastructure
Series
Annotations
Transcript
Loading...
the so we're ready now for the last hour of the
session where finds dangerously move it away from the
development plants the more applied and part see also some of the limits of the tools and much of the thinking on yeah so you
don't have any magic amazingly technologies to show on the 1st day to have good story about an organization who's trying to manage a bunch of different as points and I will try to do and what we want to what we think we might be able to do next and so
he just a bit about where come from where a be computing node in Australia we store a lot of data pattern we have a big need to be know where that comes from and I what's happened to it and where it's going well what people are doing so the things like storing metadata with the data and provenance tracking a really be deals the so what we do
at the moment is mostly make a bunch of climate data and other systems that are available for people to computing jobs on and we have pretty much starting out on sitting data ViaWeb interfaces to people to build web portals on and and basically we just going to be growing job every time and these are the points we are going to be needing to so point cloud . and we're also going to be needing to the collects really be point clouds we don't have 500 billion point clouds 600 billion point clouds in here but I still use the 7 million square kilometers and it's eventually getting covered with with what our and other other sources of Don to we also as part of our
strategy we operate in OpenStack cloud interface solely on people can fire up a virtual machine and come to be computed on and do stuff so we installed we can install point cloud operational tools for people to work on and the machine without having to draw dotted anyway and so this is
just a little bit about what we do have and this data set here the camera is pretty much the only question right that in order to Australia and there's lots of lots of other light around but it's under a bunch of different licensing conditions console collected by different state agencies and this is kind of representative of what people are asking for contracts now so it's it's relatively dense it's full waveform dada and so it gets to reasonably be reasonably quickly
and here's another example different point out that we have so this is in what are the other role in the light of data small outputs these little points are aligned the four-dimensional traces for a lot of dispersal and we 42 about to be started no idea how many points we have that but I there are many billions and this is another source of point out that we we need to better manage in some way of so we're not talking just about like
and in a with that because it's pretty but we also you know the rest of Australia have geophysical data sets that are coming in as point out so far and and take magnetics already metrics or some other stuff and he comes to us and 16 in sort of some XYZ some other bunch of attributes formats and with the light to be arbitrary all of this started in roughly the same way because of it gets used by scientists and I come from a science background and I don't have time to win 20 different tools to manage data with 1 of the other get data and use of relatively easily so the
National Computational Infrastructure as a static interoperability approach which I sort of build building a
data infrastructure platform for from all the data that we collect and this this is this is a scheme that's kind of the intonation don't look at it from that In my obstructed talked about it's just 5 because this is like the basic level of everything and I is 1 way of doing things but I'm on the other side of the coin we so I will have to be moved out of different formats unless we need to and all those people this probably no real need to we can do other things that come but
in a small natural that the entire be diagram boils down to this 1 sentences in that we have of a lot of assistance data and we want to all work relatively well together so for instance as a scientist or a data user I want the other say I have geomagnetic surveys somewhere I wanna get the latest elevation model from the latest source also that area and look at magnetics and how they relate might be something as simple as this have no shape files embedded nesting sites enormously helpful trees are just things like that there are currently not all that easy said although we have some being here week now and my we've seen a lot of mind-blowing stuff that's gonna make list of holidays so on been a
fantastic learning journey sorry on this is my compulsory australian animals like that speed words some in trouble interoperable and transdisciplinary sorry we have a bunch of animals there and they're all relatively open moralistic enjoy and don't want you the kind of lessons but not there's definitely multiple disciplines there baby you can't do anything with and by way of merging and like it they're not they not interoperable and big so we have to get to that later mn and on
we also collect point cloud data like this this is sum the structures that make kids made I made this point that the mobile fine and this is a bunch of data we're also getting to store and manage and it's open a have become really do anything with it unless you consider during parenting of transdisciplinary
task on 0 this is meant to have a pretty picture that shows books time so if you get the PDF version of the slides that shows some of the traits that that we had from working nicely with so much in current that we also have 1 pretty much the same service and we basically pulling a points thing where they go in relation to ocean currents and I'm sorry that the picture doesn't play but there is a PDF version of this and when you get a little there the thank
is such sliding sorry this is a little and animation and I think we made this all my teammates made this in about have 14 units and it's pretty cool and that's the kind of stuff that we want to be able to do all the point out that we all the again is another
Sterling animal examples of here we have In some things that some things that don't necessarily look like they can work together but if you're request the 1 that has a direction Australia and you get it
gives us all if I not the way you expected to get some sort of
society by way of exploration sort of looked at ways to put points together In New Age just files and this is an example of some of my work we wanna keep complex ways that we wanna keep on uncertainty boundary uncertainty for all the points in the dataset and we want to keep things like aircraft trajectory so that if I want regenerate those points again if I go and look at them and go well you Thompson neural messed up my computational acting justly made them chemicals would other
data necessary just coincidence photogrammetric . crowd cloud it's stored in the same age defined we can just started in there and this is it takes a little bit of time to build a file that and when you want to play with that it's really easy to just get all that tolerance and put it together on OK so that the
messy slides here but so is it is another use case on the left hand side of go on let's talk and then we have a bunch of other stuff that we generate from the latter and this is important stuff that we wanna keep in the same place as well and then we have the uncertainties associated with those different services send so at the moment we well as a scientific user of point clouds it's it's difficult to do all this stuff in the same way but using what we know
the whole without going to move it to problem put them in a database also might still follow what we basically can't do with allows for or it impractical to use text or was things like that is a more complex example of we need to make this elevation model of Australia comes from a bunch of different sources of mystery is the around pretty quickly it's the fastest moving continent on the planet and we want be and take all their life and all that the symmetry data that we've got a source the symmetry we wanna take on any of basically all of the elevation data we can get hold of nouns national together relatively frequently is currently takes a long time to build a model and you kind of trying to find a way to do it better sound this is another 1
of the problems and all Australia data pattern so we got some minor cuts and necessity and some sort of symmetry here and in this massive debt nothing say on we want be out instead of going well we need to manage this stuff over here
1 way and then all this stuff you away and stuff in other ways and it makes it really hard to then go will we kind of need to do things freeways and then managed to figure out what's in here as well harming yeah this is where
we're at at the moment point cloud data in Australia still considered difficult stuff but there's a lot of innovative things going on in terms of collecting and doing small deals that but in terms of the beta tree nodes it's it's kind of saying it is hard to manage thing because it's not a regular grid of you know files as a big and it tends to just get turned into our cells and put away and that's that's is somewhat lower cost at once every now and then but because the data are hard to use and nobody really tries to do anything with them in and here
I can some ways to put down point clouds into files that we know that but we have some as a mentioned at the start of the talk of got some specific needs in that some whatever we do with the files we need to be at track what happens to them and figure out of the process recreate infinity all around here basically know where they come from and where they're going from and we like to maintain some sort of standard compliance and and keep things fast and easy to access food and columns talking incredibly useful to us and so this is some ways that we ignore at a certain point of and these basically experiments that we need to do yet I sort of putting my abstract away from 6 months to make something cool and got waylaid along the way so time just getting started on who have done a few
things so 1 of our test here I'm always been working the with price this point and number so far far I'm quite happy with sort of come from a background of working with laser laser scanner pointing files and then the other sort of stuff to do and cost of doing pointing to database in you know and then pull back out really quickly I just made this I present book he it
takes some 45 minutes to query the whole less than 5 minutes it's it's really quick to build this picture of newly billion points and on on the left there is also of a mountain retirement and the camera and on here is 3 different classifications color separately and that's just really nice quick a going towards
some yes some things we deny before I came here very young work with this so far has led us to think will the that this is still quite a bit of can be added to make them smaller which is great and I can just keep going and figuring out what we need to do on just quickly we did
try some stiff based stuff and a natural anyone knows about the sort of of people start a library but it's a it's a way of storing points in a hasty file that's indexed by a laser pulse and looks quite nice but it's we can get to work and we need to teaching training and probably helped politics small work come in and I'm at the
end of the day we recognize that whatever we choose to do there are lots of existing workflows that use formats widely accepted and we don't break the and we don't want necessarily blow away all of our existing data that exists we need to just work in a way to keep and now this is basically just a word cloud of stuff we've been thinking about but on the outskirts of mature and so down a what enough you know I mean a lot of the stuff we've seen today and this week has been really useful for sort of guiding where going with all of this stuff but and here is a list of things to community that we highly depend on and on the yeah is cold really useful and and gray and that's the end thank
you thank you and any reaction questions yes he and when when
you say we have a we have lots of different answer which we actually want to match to the that that matches what you and what what what what did you have in mind is that like a your you want to do and the big dots analysis where you're never sure what's going to come out um or you only have in mind what you wanted to to mention to each other on so that in terms of building like national elevation molar intensive some scientists coming on and writing down the engines also because we we basically so that and we have no idea what users are going to do so in a way where would kind of mandated to make access to the data as easy as possible so and we want to enable so those things that haven't been easy to do in the past like there is I think in existence of study where some scientists in Australia validated spot 5 tree cover metrics using a 1 liner and was a big study and it was it was not so easy for them to do and in a way we would like to be out get people to reproduce that just on anywhere in the country just grab a chunk of data they began a polygon and I will get the the spot immediately the ladder and was do stuff with and without having to become domain experts and how to access and analyze water or other point on other questions yeah look at moment and thanks to anonymous and this closes the session of right now and then we have the closing plenary keynote and closing session and thank you
Software developer
Programmable read-only memory
Point (geometry)
Point cloud
Limit (category theory)
Mereology
Open set
Computer animation
Pi
Meeting/Interview
Computational science
Moving average
Hill differential equation
Gamma function
Point (geometry)
Service (economics)
Point (geometry)
Video tracking
Point cloud
Bit
Euler angles
Open set
Metadata
Computer animation
Mathematics
Web service
Vertex (graph theory)
Computational science
Self-organization
Pattern language
Subtraction
Task (computing)
Point (geometry)
Building
Multiplication sign
Source code
Process modeling
Virtual machine
Point cloud
Mereology
Virtual reality
Strategy game
Operator (mathematics)
Personal digital assistant
Electronic meeting system
Square number
Integrated development environment
Vertex (graph theory)
output
Physical system
World Wide Web Consortium
Installation art
Metropolitan area network
Web portal
Point (geometry)
Moment (mathematics)
Interface (computing)
Supercomputer
Subset
Process (computing)
Computer animation
Web service
System programming
Point (geometry)
State of matter
Point (geometry)
Source code
Process modeling
Design by contract
Bit
Manifold
Set (mathematics)
Function (mathematics)
Tracing (software)
Population density
Population density
Computer animation
Order (biology)
Representation (politics)
Dispersion (chemistry)
Integrated development environment
Waveform
Right angle
Subtraction
Condition number
Point (geometry)
File format
Multiplication sign
Point (geometry)
Process modeling
Set (mathematics)
Metric tensor
Attribute grammar
Fluid statics
Computer animation
Moving average
Integrated development environment
Quicksort
Numbering scheme
User interface
Scientific modelling
Source code
Archaeological field survey
Shape (magazine)
Infinity
Emulation
Natural number
Energy level
Integrated development environment
Diagram
Computing platform
Area
Metropolitan area network
Sine
File format
Interior (topology)
Electronic mailing list
Instance (computer science)
Entire function
Computer animation
Network topology
Personal digital assistant
Computing platform
Website
Boiling point
Domain name
Wide area network
Point (geometry)
Word
Summation
Computer animation
Inheritance (object-oriented programming)
Point (geometry)
Point cloud
Point cloud
Data structure
Open set
Data type
Point (geometry)
Ocean current
Slide rule
Ocean current
Theory of relativity
Multiplication sign
Point (geometry)
Revision control
Web service
Computer animation
5 (number)
Quicksort
Units of measurement
Task (computing)
Probability density function
Computer animation
Direction (geometry)
5 (number)
Point (geometry)
Computer animation
Computer file
Multiplication sign
Point (geometry)
Trajectory
Boundary value problem
Bit
Point cloud
Quicksort
Trajectory
Surface
Slide rule
Inheritance (object-oriented programming)
Multiplication sign
Scientific modelling
Mobile Web
Computer-generated imagery
Source code
Point cloud
Horizon
Data model
Web service
Video game
Database
Row (database)
Aerodynamics
Subtraction
Source code
Point (geometry)
Moment (mathematics)
Complex (psychology)
Mereology
Population density
Computer animation
Symmetry (physics)
Personal digital assistant
Source code
Summation
Computer animation
Symmetry (physics)
Modal logic
Pattern language
Mereology
Quicksort
Mass
Film editing
Trail
Beta function
Computer file
Multiplication sign
File format
Point cloud
Term (mathematics)
Network topology
Data storage device
ASCII
Standard deviation
Variety (linguistics)
Process modeling
Point (geometry)
Archaeological field survey
Moment (mathematics)
Metadata
Infinity
Point cloud
Motion capture
Transformation (genetics)
Computer animation
Network topology
Web service
Vertex (graph theory)
Software testing
Quicksort
Data management
Electric current
Point (geometry)
Query language
Computer file
Structural load
Geometry
Server (computing)
Point (geometry)
Interior (topology)
Point cloud
Usability
Number
Computer animation
Graph coloring
Database
Software testing
Software testing
Quicksort
Subtraction
Point (geometry)
Computer file
Waveform
File format
Laser
Area
Wave packet
CNN
Electronic meeting system
Subject indexing
Library (computing)
Metropolitan area network
Point (geometry)
Computer file
Coma Berenices
Point cloud
Bit
Group theory
Existence
Maxima and minima
Computer animation
Web service
Data compression
Software testing
Quicksort
Pulse (signal processing)
Library (computing)
Query language
Spacetime
Numbering scheme
File format
Geometry
Server (computing)
Point (geometry)
Computer file
File format
Curve
Electronic mailing list
Point cloud
Point cloud
Mereology
Group theory
Open set
Thread (computing)
Word
Word
Computer animation
Subject indexing
Quicksort
output
Capability Maturity Model
Covering space
Point (geometry)
Domain name
Dot product
Building
Existence
Observational study
Closed set
Polygon
Moment (mathematics)
Expert system
Mathematical analysis
Water vapor
Metric tensor
Computer animation
Network topology
Term (mathematics)
Subtraction
Loading...
Feedback

Timings

  455 ms - page object

Version

AV-Portal 3.8.2 (0bb840d79881f4e1b2f2d6f66c37060441d4bb2e)