Bestand wählen

24-hr Latency End to End Data Processing Using Open Source Technologies for the Airborne Snow Observatory

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Erkannte Entitäten
go ahead and get started so try keep a song scandal I guess but I notice when I just look at the size hadn't actually update the title the slides but really when you talked about today is a demonstration mission that NASA's doing on the job laboratory over the Sierra Nevadas and that work that's going on and how we you're trying to deliver of some products to California Department water power to work water resources and also water managers in along the differ dams along this hearing about us and minus foreigners and multi multi of Estonia's with me to he's gonna talk a little bit row quickly but some maps work creating and some parts were producing but I'm a talk more about the project itself have actually been at JPL for about models going on 14 years now doing a lot of different er science and planetary science projects the doing data processing in general and background is actually computer science and not a remote sensing expert and tend to learn some snow signs of working on some these projects but if you have a real tacky questions about snow probably gloss over a little bit and then get into some you someone else on the team and the this mission was actually very I think it's really stretches so hopefully you'll enjoy what I'm about to talk to me so in California water is a big thing and and all of our water comes from the mountains and understand the snowpack and there's you know things like water conveyance there's also things like storage and just you know the mere crops and at the in the day and these drive the economy of California so it's a big deal and I know it's a big deal because we're having a big drought and in this demonstration mission it actually put a lot of focus on doing so because of the drought and really this this mission is just at this point a demonstration of the platform and what we could do in the future with snow and work towards a spaceborne instrument so were right now in the 2nd year but were 0 3 emission but we're getting there OK
so tradition if you are concerned about as houses snowpack measure quote by matter 1910 the limit can have people out in the cell like that in skis next elucidated today and took measurements like that you know the pole because if you were in southern California promises Nellie times and services go you'll you'll see this in the sentences with chronicle and I know some these guys know they're doing this still franker his on those guys sitting out from the poles is actually our Wickles poles are is actually on our team and helping to transform the way we we do the science and more recently here others in-situ measurements that happened throughout this year in about you know California probably 1 of the best uh monitor resources and right now and from the CERN about us but even that is low level troublesome when you're trying to really drive the economy of California and really understand the snowpack because what you get with somebody's measurements if you look at the tony River Basin I get a picture that looks like this and this is 36 100 times expanded and you can probably see or maybe you can see here and we will right off the difference and you know what so this is the unit of something I don't wanna reason which feeds all stuff so we wanna do better they want a better what they really want to see is something like this and there was a 15 year resolution on on in 24 hours at this scale which each row across the written all over this year and working to do that like as were in the 2nd year and without a lot success actually in a 1st year we had some really great success but we were actually able to operationally plug-in to Hagerty reservoir and change the measurements and more actively than than with the anomaly provided and the forecast model so here you can see up in the red line you can see what they're estimating in terms of water that was gonna come into the reservoir or without a cell and then what the actuals work and then what they were doing with with with their cell so we were able to improve the measurements by a factor of 2 and actually late in the season of they lost their normal more data stream and exodus used in so data directly to to decide how much water was some of the and was 1 of the 1st year cause they were actually releasing some water this year was more about how much water you see how much is gonna be preserved so this was 1st year results so operationally we deliver now how are we doing this well there's 2 types of instruments needed to be done this do the sets of observations observations and understand the snow pack much better 1 is an imaging spectrometer that you see over here on the right which is albedo and understanding of how fast the snow was in and out and the other is a lighter instrument sitting there on the left and right using some body instruments and this 1 is used on a bunch of other NASA our projects the real sensor over there on the left is actually a a brand new sensor that provides us with 3 D scanning wider and wider in other words we can actually capture outcrops of the mountains and we get what about let's see we get about 1 . 5 meter resolution down the 3 meter resolution something within the Pauli River Basin and this session at the lattice and actually change within the 2nd year of the and the lower using the 1st year wasn't up to snuff so we went with this 1 and this 1 was great and suck part of my story here to tell you about a demonstration mission and working with research and technology development is everything changes in the 1st year was utterly different this year was a new instrument next year we should have the same platform to prove this technology and but we're going to expand where we're going with this particular instrument and in order to do the three-D scanners produce 3 models of the area under has to standards that are offset so it has the passing through these basins it can catch those over crops and overhangs and find out where the water is the snow is actually depositing where the water is at the basis of so but from the 1st year to the 2nd year that end up producing about 3 times a day that were expecting so we had a chamber system architecture change the algorithms that we had in basically change everything from yeah 1 which we approved over able to do it for the 1st year 2 so was also in and 1 of them esos 40 coal because it actually gets in the news a lot and and because we've accomplish our goal of 1st year sort of but not the long-term goal but the short-term goal and this makes me happy because I can explain to my mama just boarded pointer to article that really gets around all the science stuff and really just tells you are letter instruments trying to measure how much snow is on the mound and by doing 2 types of measurements 1 when there is no snow and 1 when there is no and then the that imaging spectrometer is really telling us and how fast the snow will melt what's the refractivity how much rate of energy is being absorbed into it and these are the 2 things that you need to to understand and understand quickly I what elements
the 1 thing I miss their own at that on the other slide was that these instruments in the measurements were taking our within 10 centimeters resolution accuracy and typically our star resolution our accuracy is actually on the 3 centimeters but and that's a were able to make some these measurements and providers the things that you
the team itself is made of 3 teams actually there's a field team that does so we can do ground truthing takeout field measurements throughout the gym digs no pets which I haven't done yet but will do next year the sounds cool looks fundamental I probably actually do it and then we have a competing that sits behind all these computers and does all this processing a remotely and just have fun that way and see little that a pictures come out and there is a flight team which is really cool they're sitting on the plane turning instruments on and off and taking pictures and and deciding when there's go no-go decisions and then moving at data around by taking raw data breaks off a plane and delivering them to that the system that we built a you'll see here our competing lead is actually Doctor Chris man and he was on the agenda initially come talk I'm actually leaving the computer operations and in the 2nd hour have done most of the computing stuff and was easier on the team he works on the CAZy the imaging spectrum beside and so our job is actually really
simple we take a step back we just don't need to lose the bits allowing to keep calm and not worry about anything else if we didn't make the 24 hour time case then it really wasn't a big deal because it is a demonstration mission is just a proof of concept of to show that word we can do this and that to turn it into a spaceport instrument but now we also need to rapidly integrate science algorithms and and the science algorithms tend to change as were going along get these measurements are off or there's something need to tweak often harassed to I we take delivery from the science team and there it is of into our pipelines so we need to be able do that quickly and then reproduce the data quickly and we need to ensure reproducibility of data processing and capture the provenance like is that we actually end up having what we call processing campaigns and right now a actually reproducing the other day that we took during the snow melt seasonal told me that to provide higher level accuracy and because there are sometimes constraints that that reduce the level of accuracy but don't affect decision-makers but do offense that affect science scientific quality and and then the other we have is just to deliver and transfer data both within the system and also to people and you know deliver down the JPL on fumble little things like works I have a little picture there coming up who we also create decision support products and get them out to California be are and John C. that damn manager and actually symbols you Berkshire gene selection validating that we can and should do this and these are affecting the decisions that being made and last but not least we obtain the scene since we're sitting behind computers we have a bunch of extra time does have some fun and cracked some jokes and you try to do something in 24 hours and yet you need to have some of to it so actually birth data who went on our 1st operational or missions this year for but for the 2nd year from the campaign who so we have 4
campaigns so far no 1 was a snow off campaign in 2012 that really just focus on the Twomey and some of the and the pottery and never Colorado which I'm not really talk about the year now we had a full snow on campaign in the 1st year and we were able to prove the 24 hour concept and get it done and at that point it became well what do we do next and better accuracy so that's a new lattice unit that's everything changed but in the 1st year we were able to prove that and we do not to start out out there during the seasons and the 1st few weeks she failed for about 3 or 4 weeks you know they got past 4 year 50 hours and hours like our done and let's see what's wrong the streams are ribbons of update what's going on and looked at some new harbor to the system that the the then we recently have another snowing campaign to revisit the upon real River basin now we actually had approved then we could do this and other parts of the scared about us and what that would take and so we visited the the Merced the Kings Canyon the lakes this year and we have that data and we produce sound difference maps but we were doing that in 24 hours because we hadn't done this off campaign and those areas and that's what we're doing right now there actually flying around as I'm speaking in those areas capturing a digital elevation models so we can produce those we maps and they're are actually doing most of us here about some extra reading you this operationally within 24 hours and which will be found which probably will mean 2 more planes and a little more the
so out of all this is some very basic Hannibal products and snow water equivalent down to 15 their resolution they're releasing to the public and the the water members snow depth at the same resolution and diems and is DSNs actually within the DMZ we have of this area are down to about 1 . 5 meter 3 meter our actually gonna produce of even higher quality 1 by just taking the differences amongst each of the weekly campaigns and adding them together to create a really high fidelity DM of all these hearing about us as the imaging spectrometer of course yeah they'll be a measurement we have snow cover we brain size and reinforcing how much energy has been absorbed into the snow but beyond this and there is some zonal statistics that happen across all these products and Muzzy we'll talk a little about that and some open-source tools that were using to produce those products and but how do we really
do this well manifolds and you think it's in California and there's a bunch of network connectivity and it would be easy just like show the state up to the cloud in Galveston all quickly and would be great 24 hours yes no problem now this is not and the network and activity analysis horrible and this they're about to transfer 500 gigabytes of data along that line we were talking about I think uh 0 somewhere on the order of 3 or 4 days and because it's well that's what it would take this year they upgraded to a T 1 line and that facility but uh that's still wasn't going to work are cut so what we end up doing is just building this what we call the mobile compute system cluster of computers leadership it up as a box up to this year the better what's Research Laboratory symbols pictures you saw there was actually sitting there in the 1st operational mission but lots of times when we're doing these slides I've actually sitting back at JPL 1 occurring at or at my and nieces part birthday party logging in the IIsi years into the SSH to see what's going on and so I said it's a little more
complicated and we do some really random stuff and the random stuff is like and being its narrow but also having all these bricks that go get shipped around and critic you our systems of action over these bricks or bricks are and you have a flight team that's driving through the snow and you have people deliver you flight logs that originally look like some scribble text in trying to process data that way which then turned into a digital format but it was really cool and then you have some the mammoth airport sitting there you have GPL so all this the context of war this is going on how this is happening in 24 hours includes all these areas includes people landing on the plane taking this off the plane transferring them when they're sort of hypoxic over to Sierra Nevada clocks Research Laboratory of plugging them into a computer system and that's what you remotely yeah it so
we thankfully needed or actually Joe Morgan who was on a team thankfully name this 24 hour thing a rodeo and he sent out this image of a like great that sounds cool because all especially need some we're going for it and he sent spectra was really cool and he actually he's been doing a lot easier warm projects and in the beginning he didn't really think we can get it done in 24 hours and when we start because he was like well you guys actually the 2nd thing was gonna can happen and because in there were missions that he's done in the past with NASA the way we've done and is collect the data bring it back to JPL or bring back to some of the facility 3 Slater process that data there's not as many constraints there's more data but the availability of so what are we really
talking about were take talking about taking Roberts down what mass recall level 0 and taking all the way up to level 4 or 5 doing orthorectification cutting down trees and uh due locating the data and doing some atmosphere correction in the middle of a copy itself but the and everyone just cares about some nice pretty maps and some nice little statistics down the bottom a list the water matches to the centers care about what more but what Amanda's once the stuff like this and I here is a really high level picture of our workflow and and the algorithms that are that we have within the system this is actually just the automatic that the automated portion or at least the things we get out of me but in dealing with this like I said we had a vendor-supplied light our and you actually end up having to use the software the software requires a person and yes there is no way descriptive were actually working with them to use be able offer the right now to for a QA QC it requires some to go in there and remove points that are outside and do some really cool stuff and deal with the format and to get to the point where you have last files and then it runs through these types of pipelines and disco picture here if you're asked if you ask me why this stuff was green yellow and red and we took this as snapshots during the year and green we were good everything was happening break in the 1st year Guillemette 0 were sort of you to figure out what the hell's wrong here and red that just failed and we change these things the model the process that way so right now where we stand as we've made the in this of promotor path fully automated to the point where I log and dire in sort of ask them you can ask about a reports back on what lines have been processed tell faster more with the pipeline with power them there and and that's really easy I just put at the at the in the dead the data gets delivered to the MCS and butterfly in file and everything happens in Montfort remotely like it said the letter prof such a little bit more at this point it requires humans it's based upon the humans are required for to couple 2 reasons that vendor software but also because there's a time-constrained terms so when you can get base-station information to produce these products within 24 hours with the accuracy you need and sometimes we deploy our own base stations but we can get the coverage we need so were dependent on the base station information showing up at a given time and also the ephemeris data that will help the accuracy or positional accuracy of our data actually gets released at 8 PM and I know that because I'm sitting there at 8 PM waiting and it's random and of it so that such window between 8 PM and 9 PM and that's when you can get the 1st non-scientific quality ephemeris data anyways so it's a little human loop waiting for stuff to happen and that stocks and we don't want that but were there and a a lot of this form so Neruda handle on I the faster to an error handle some of the stuff we use some open source software I love Apache admin commit across all our products a data management system I think that we use is a color patch unity ask about it later essentially helps us wrap all these algorithms in in the written much faster and it's not like Hadoop a MapReduce were not rewritings these algorithms these people are delivering them to us like scientists on a team they're making updates of it's more about how do we call them that we give them data and how do we Marshall balls out on 2 different the resources so the is that Pat Apache Tika is content analysis to get it out actually helps us extract all the metadata from these files and then push it offered to us on that so you can search and and the due process in that we have no typical that difficult in there so we can monitor everything the IIsi because Raul Iousy geeks and trying to something operationally communication is a big key so we publish everything out and through to go and
now mothers can talk about some simple examples the OK and so I wanna talk about the true juries will guide guided and the end of the the of process 1 of them is the analysis acidic
so you see on the that side is the duty of comes from the on the process it says suis snow water equivalent so the pixel value shows how much a snow is on each of these areas and this is the base and shape all so what I'm doing just sympos analysis to seduce the open source to the library to you will all of these small polygons find the pixel from the rest of the team each polygon and then do some simple as that is sick and like and mean value max somebody you and send those as a takes takes log to the that the California Water Resource manager so they they know for each of his polygon was slow how much noise-generating that's and that's the 1 processing the other 1 is to
alter my be able to make the maps so as you can see these other items I'm using in my Python code is the base map of that area of its armor bottom means that religion like is skin and on the nor arrow I generated those from the q g is its actual data that's gonna be change every time every afternoon flight this the color tables to make the make the like the duty from conifold food and this is the and the border after the 2 basis of the code them or to they the code but that but the program is doing it basically how all of desire to and kind of try to make the map
actually we have to be deal for that the CEO of so we snow water equivalent so as you can see these are the date the
date is changing and all of these maps being generated after each flight 24 hours so as soon as the fly lands that the whole process comes to the the school maps that shows how much of water comes from each area all of the codons and
detainees writing is going to be open source for the geospatial part and its is like topic the protocol due time is a a cookbook for all the jails set of examples using open source GIS tools
so if you wanna get some of our data and I say some of that because we haven't released like the last files markers were not a distribution center but at some point they would be available on through it like an inside sea once this mission gets a lot further out and that the geo test for the area of doing these flights are actually get pushed automatically after a website there there this we in the albedo files and you can have fun with them and do what you want and which and the weight at have it at the end of it's just a so that g go Mossadegh of so here we are now
this is what we're were studying this year as and next year this is where we wanna be all the Sierra
Nevadas and and that's and this is sort of were talking about how this can extend the via a spaceport instrument operating a new program or a enough delivering this operation to somebody else is sort of in the mix right now and since we've had some successes and
In order to get there we have to think of our complete system as a building block and right now that building block was between JPL and this year what about a plots Research Laboratory at some points and we will try to get onto the plane everyone asking that I like it said that the main thing there is just 1 meeting at the delivery of other parts of the data and the fact that some this of supplied suffer requires a band of the loop so putting on the plane is not a big thing right now at some point maybe would package input put on a cloud if network latency was was low in that area but at some point and In January actually is so is going to go down to chile it may replicate itself so we can have 1 to run in the Sierra Nevadas so but they're gonna do some experiment we actually have an ventures proposal to do this in the Himalayas and and we'll find out whether or not we're doing that in October or end of October and that will be called to this hydrogen the Paul twice but to go there aboard hands of the great and that'll change sort of the mission primers because in the poll Dorado base stations so being able to get that will have to also deploy all over on base stations and work with the local people figure out how we produce the same quality data at the same time frame but that's the goal of the brief
summary and it is sort of a big data scenario I talked about it like fight fight under gigabytes of data to but 1 terrabytes a religious depends on what your network latency is to be able to produce that in 24 hours and basically it's just comes down to where is your compute worry pushing it to that's why billet as a building block and what your delivery constraints if we were working within 24 hours it wouldn't really matter we could do this over a week would be a big deal and that's called the commuting actually does more than just sit behind we marital delivery of dust and movement hardware and system configuration system engineering and like I said were preparing us for the future of either space this instrument or doing that EVS proposal of in last and is so currently does provide a market picture so water and the Sierra Nevadas and will continue do so and will prove that next year and across all is seared about us like we've already done a disaster for me any questions about the presentation the
arm of the others my my e-mail is pretty easy pure Maris my name that you kill the mouse so go the creatine Chris Mammon takes a lot of questions um Tom painter is the main scientists there that cater to the Council were flying we actually we got some pictures about what were flying there's not too many followers because I don't know when we had a short account and also just fans of fine happy deduce to a at that point I pretend like I'm a plane so it's sort of funny at some point you may find it funny either there is a website and like as it were heavily in Apache attack Apache together so that is up on the user less Assis about how we're using those tools I would be happy to talk about it connected
Service provider
Streaming <Kommunikationstechnik>
Vorzeichen <Mathematik>
Gruppe <Mathematik>
Trennschärfe <Statistik>
Statistische Analyse
Metropolitan area network
Shape <Informatik>
Gebäude <Mathematik>
Kontextbezogenes System
Dienst <Informatik>
Rechter Winkel
Ordnung <Mathematik>
Tabelle <Informatik>
Lesen <Datenverarbeitung>
Spektralzerlegung <Physik>
Konfigurator <Softwaresystem>
Räumliche Anordnung
Open Source
Fächer <Mathematik>
Reelle Zahl
Topologische Mannigfaltigkeit
Protokoll <Datenverarbeitungssystem>
Open Source
Elektronische Publikation
Patch <Software>
Wort <Informatik>
Prozess <Physik>
Atomarität <Informatik>
Element <Mathematik>
Einheit <Mathematik>
Prozess <Informatik>
LASER <Mikrocomputer>
Mixed Reality
Zentrische Streckung
Nichtlinearer Operator
Prozess <Informatik>
Physikalischer Effekt
Speicher <Informatik>
Plot <Graphische Darstellung>
Arithmetisches Mittel
Projektive Ebene
Web Site
Gewicht <Mathematik>
Virtuelle Maschine
Zellularer Automat
Kombinatorische Gruppentheorie
Inverser Limes
Zeiger <Informatik>
Speicher <Informatik>
Bildgebendes Verfahren
Leistung <Physik>
Einfach zusammenhängender Raum
Physikalisches System
Mapping <Computergraphik>


Formale Metadaten

Titel 24-hr Latency End to End Data Processing Using Open Source Technologies for the Airborne Snow Observatory
Serientitel FOSS4G 2014 Portland
Autor Ramirez, Paul
Lizenz CC-Namensnennung 3.0 Deutschland:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/31707
Herausgeber FOSS4G, Open Source Geospatial Foundation (OSGeo)
Erscheinungsjahr 2014
Sprache Englisch
Produzent Foss4G
Open Source Geospatial Foundation (OSGeo)
Produktionsjahr 2014
Produktionsort Portland, Oregon, United States of America

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract JPL's Airborne Snow Observatory is an integrated imaging spectrometer and scanning LIDAR for measuring mountain snow albedo, snow depth/snow water equivalent, and ice height (once exposed), led by PI Dr. Tom Painter. The team recently wrapped our second "Snow On" campaign where over a course of 3 months, we flew the Tuolumne River Basin, Sierra Nevada, California above the O'Shaughnessy Dam of the Hetch Hetchy reservoir; focusing initial on the Tuolumne, and then moving to weekly flights over the Uncompahgre Basin, Colorado.To meet the needs of its customers including Water Resource managers who are keenly interested in Snow melt, the ASO team had to develop and end to end 24 hour latency capability for processing spectrometer and LIDAR data from Level 0 to Level 4 products. Fondly referring to these processing campaigns as "rodeos" the team rapidly constructed a Big Data open source data processing system at minimal cost and risk that not only met our processing demands, but taught the entire team many lessons about remote sensing of snow and dust properties, algorithm integration, the relationship between computer scientists, and snow hydrologist; flight and engineering teams, geographers, and most importantly lessons about camaraderie that will engender highly innovative and rapid data systems development, and quality science products for years to come.Chris Mattmann, Paul Ramirez, and Cameron Goodale for the ASO project will present this talk and will detail the story of the Compute processing capability on behalf of the larger team, highlighting contributions of its key members along the way. We will cover the blending of open source technologies and proprietary software packages that have helped us attain our goals and discuss areas that we are actively investigating to expand our use of open source.
Schlagwörter snow
remote sensing
point clouds
open source
decision support

Ähnliche Filme