OnEarth: NASA's Boundless Solution to Rapidly Serving Geographic Imagery
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 188 | |
| Autor | ||
| Lizenz | CC-Namensnennung 3.0 Deutschland: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/31677 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache | ||
| Produzent | ||
| Produktionsjahr | 2014 | |
| Produktionsort | Portland, Oregon, United States of America |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
00:00
HilfesystemOpen SourceSchlüsselverwaltungInformationssystemKonfigurationsraumEinflussgrößeAuflösungsvermögenWeb ServicesDifferenteDateiverwaltungDateiformatTesselationTranslation <Mathematik>PhysikalismusProdukt <Mathematik>PackprogrammAggregatzustandBitGrenzschichtablösungServerMereologieModulare ProgrammierungKontextbezogenes SystemCoxeter-GruppeObjektorientierte ProgrammiersprachePunktInformationsspeicherungKollaboration <Informatik>BenutzerbeteiligungBildauflösungLuenberger-BeobachterSatellitensystemPhysikalisches SystemVerschlingungElektronische PublikationStandardabweichungMaßerweiterungProjektive EbeneCachingSchnittmengeSoftwareentwicklerDatenkompressionGoogolGebäude <Mathematik>Overhead <Kommunikationstechnik>Komponente <Software>Mosaicing <Bildverarbeitung>Virtuelle MaschineClientBildverarbeitungMAPKommunikationsprotokollOrdnung <Mathematik>ZoomSoftwareMeterTreiber <Programm>TypentheorieParkettierungWeb-DesignerMultiplikationsoperatorRuhmasseErhaltungssatzExogene VariableGüte der AnpassungInformationSkalarproduktraumOntologie <Wissensverarbeitung>AbfrageUmwandlungsenthalpieCASE <Informatik>DatenfeldMetrisches SystemFontWort <Informatik>BenutzerschnittstellenverwaltungssystemElektronischer ProgrammführerQuantisierung <Physik>Figurierte ZahlInformation RetrievalAnalogieschlussUnrundheitResiduumKartesische KoordinatenMagnetbandlaufwerkARM <Computerarchitektur>Gruppe <Mathematik>MultiplikationKlasse <Mathematik>BinärcodeGesetz <Physik>Nichtlinearer OperatorInformationsüberlastungTurm <Mathematik>QuaderMusterspracheVakuumpolarisationKette <Mathematik>Sichtenkonzept
08:41
BestimmtheitsmaßClientMAPProdukt <Mathematik>Prozess <Informatik>BitInformationLeistungsbewertungProjektive EbeneServerElektronische PublikationKonfigurationsraumElement <Gruppentheorie>MeterOpen SourceComputerspielPhysikalisches SystemGraphfärbungZahlenbereichRechter WinkelGenerator <Informatik>BinärdatenLesen <Datenverarbeitung>Gewicht <Ausgleichsrechnung>KommunikationsprotokollGrenzschichtablösungMereologieDateiformatTesselationRandverteilungCASE <Informatik>IntegralSchar <Mathematik>Automatische IndexierungVorzeichen <Mathematik>PaarvergleichAuflösungsvermögent-TestMinimumRechenwerkp-BlockEinflussgrößeStichprobenumfangDimensionsanalyseUmwandlungsenthalpieÖkonometrieMultiplikationsoperatorMaßerweiterungFigurierte ZahlBilddatenbankKartesische KoordinatenSpezialrechnerSkriptspracheMetadatenDatenkompressionEnergiedichteSoftwareSoftwareentwicklerTurm <Mathematik>Translation <Mathematik>Wort <Informatik>E-MailBinärcodeSingularität <Mathematik>Zeiger <Informatik>Stabilitätstheorie <Logik>PunktCachingAbstraktionsebenePlug inModulare ProgrammierungComputergraphikMetrisches SystemStrömungsrichtungZweiLoginAdditionWeg <Topologie>Computeranimation
17:22
SichtenkonzeptClientTesselationSoftwareSharewareComputeranimation
17:57
SichtenkonzeptVarietät <Mathematik>DatenkompressionSoftwareClientWeb ServicesFrequenzVorzeichen <Mathematik>SystemaufrufMobiles InternetLineare AbbildungPartikelsystemTouchscreenMinkowski-MetrikStichprobenumfangRückkopplungProjektive EbeneVektorraumUmwandlungsenthalpieMailing-ListeLuenberger-BeobachterZahlenbereichMAPInformationOpen SourceCoxeter-GruppeMomentenproblemKartesische KoordinatenTypentheoriePunktSondierungWeb SiteGrenzschichtablösungGruppe <Mathematik>Offene MengeStandardabweichungAuflösungsvermögenTesselationSoftwareentwicklerZeitreihenanalyseVersionsverwaltungServerVerschlingungFreewareKrümmungsmaßPackprogrammProdukt <Mathematik>OrtsoperatorSkriptspracheSatellitensystemBitmap-GraphikHilfesystemBitQuelle <Physik>Multiplikationsoperator
23:21
VerschlingungWeb-SeitePhysikalisches SystemAdressraumPunktInhalt <Mathematik>E-MailVorlesung/Konferenz
24:05
ViewerWort <Informatik>MultiplikationsoperatorPunktAttributierte GrammatikDifferenteAggregatzustandElektronische PublikationQuick-SortGewicht <Ausgleichsrechnung>StandardabweichungGenerator <Informatik>Prozess <Informatik>TesselationMailing-ListeServerClientMaßerweiterungGraphfärbungRechter WinkelComputerspielPhysikalische TheorieSichtenkonzeptKommunikationsprotokollGrenzschichtablösungMooresches GesetzBildauflösungGruppe <Mathematik>ParametersystemProdukt <Mathematik>MAPInverser LimesExpertensystemAnalysisKonzentrizitätZeitstempelDateiformatKoordinatenGanze FunktionVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:00
Good afternoon. Thanks for coming. My name's Joe Roberts. I'm from the NASA Jet Propulsion Laboratory, just like the presenters before me, or JPL. It's managed by the California Institute of Technology. I work on a team called the Global Imagery Browse Services, or GIBS. It's a collaboration between JPL and also the NASA Goddard Flight Center.
00:24
Today I'm going to talk about On Earth. Just a brief outline, I'm going to explain what On Earth actually is, a little more details about GIBS, get into some more of the technical stuff about On Earth, a little look at performance and metrics, and if there's time, maybe
00:44
do a little preview of some of the client applications that you could build using the On Earth server. At the end, I'm going to just briefly go over our open source project. So what is On Earth? It's basically
01:03
an open source image processing and server software package. In other words, it's a tile server. It's intended to provide an out-of-the-box solution for generating geographic imagery. It's a little bit more than just a tile server. We also include components to
01:21
build tile pyramids from a global mosaic and then store those pyramids in this image archive. So the intent of the archive is to be able to serve tiles to some client application using standard web protocols. For our case, we use WMTS, Tile WMS, or KML. Tile WMS is actually
01:45
an extension of WMS. It was developed at JPL a while back, a long time ago. It's just a way of creating a predefined set of cache tiles. So when you do a WMS request, you already
02:01
retrieve a set of cache tiles. And I guess the key selling point of On Earth is that it's designed to be lightweight and very performance driven. We use this special file format called the Metal Raster Format, or MRF. It's essentially not an image format per se. It's an image
02:24
container format. And we use this because it has very fast access and we're not limited by image size or resolution. We use a lot of GDAL to generate the MRF. We have this MRF
02:40
custom built driver that we use to run GDAL translate from one image format, JPEG or PNG or whatever, and be able to generate these tile pyramids into the MRF format. For the actual image server, it's essentially just an Apache module. We call it Mod On Earth.
03:01
And on top of that, we have some configuration tools to just kind of help set up the server. Just a little bit of background about On Earth. It's been around for a while. Its development began originally at JPL in the early 2000s. The lead developer at that time, Luchen,
03:24
he's now over at Esri. But luckily being an open source project, we still get some pretty good contributions from all around. It was formerly known as a Tiled WMS Server. This is back at the time we were just using Tiled WMS, but now we've expanded to
03:40
more commonly used WMTS specification. Some of the other things On Earth has been used for includes planetarium shows at public museums. It was the first image server to serve out 15 meter global Landsat data or imagery. And for those of you who remember NASA whirlwind,
04:03
it was kind of a predecessor to Google Earth. On Earth was used as the actual image server for that application as well. Outside of planet Earth, I guess, we have an image server for Mars and the moon. And respectively, we call that On Mars and On Moon.
04:22
PODAC, which is the Physical Oceanography Distributed Active Archive Center. It's based at JPL. They have this state of the oceans tool to present imagery of all the world's oceans. That's also built on this On Earth server. Just a couple years ago, we started this project
04:41
Gibbs back in 2011. And On Earth was selected as the primary image server for that project as well. And on top of Gibbs, the client application most commonly used is NASA Worldview. And finally, just last year, we released the software
05:04
in open source on GitHub. So it's really available. Anyone could go up and access and download the software. Okay. So a little bit more about Gibbs. So NASA, as you all know, has several Earth observation satellites constantly taking measurements of the Earth.
05:23
And we have this really rich and vast image archive. So it's not really useful unless it's available to general public scientists and so forth. So the purpose of Gibbs is to provide the full resolution image archive and access services to get to all these different data products. Currently, there's about over 75 global satellite products. Most of them are available
05:46
within four hours. It's part of this Earth observation system data and information system ESDIS. And because it's available to the public, we need something that's really scalable,
06:02
fast, responsive, and so forth. So for more information, there's a link down there. We also have another presentation, Gibbs-related, at 4 p.m. that's going to provide some more of the context and historical background of Gibbs. Oops. So I guess one of the key selling points
06:26
of on Earth is speed. So the key to the speed and storage is this metaraster format that I mentioned earlier. So most of the overhead in actually getting to the imagery,
06:41
or not most, but much of the overhead is having to do with the file system. If you have images sitting on a machine somewhere, just a bunch of PNGs or millions of tiles just sitting on file folders, that introduces some file system latency. So to reduce that, we put everything
07:01
into one big data file. So when you're accessing the data, you just have to go to one point without any heavy file system operations. This has been around before. I mean, nowadays there's different image file formats or container formats for tile pyramids such as MBTiles and Geopackage.
07:23
So it's been around before that. We didn't really consider anything else during that time. Let's see what else. So with MRF, we can support, oops, sorry about that. With MRF, we can support
07:46
multiple projections. For Gibbs, we use geographic lat-long, arctic and antarctic polar stereographic, and also Webmercator for Google Maps, Bing Maps, and so forth. Some of the compression types used, this is for when you pull out the imagery, you could
08:03
pull out either JPEG, PNGs, or even just binary data. I don't know of any web clients that actually use TIFF, but primarily it's going to be JPEG or PNG. I mentioned earlier there's a driver for GDAL that we use to generate the MRFs. And to do that, we basically just run GDAL
08:29
Translate. We take an image composite and just convert that to an MRF. And on top of that, we run some GDAL commands to build up the different zoom levels of that pyramid in order to get the final product. MRF is not itself by itself a single file. It's actually
08:45
composed of three different files. The first being the MRF header is just an XML based file containing the metadata for the MRF itself. There's also this index file, which is basically just a lookup to somewhere in the binary data file, which contains all the images.
09:10
So in the metadata file, it's pretty basic actually. It contains just the full base
09:20
resolution, some of the compression information, the size of the tiles. We normally use 512 by 512. There's also additional elements besides this. You can have information about the color maps and other projection information as well.
09:42
So the data file is pretty simple actually. It's just a bunch of JPEG images or PNGs stitched together side to side, starting from the full base resolution at the bottom of the pyramid and moving upward. Each block itself is the actual JPEG image. So once you pull
10:03
the data out, you actually get back that JPEG or PNG. There are some drawbacks to this. Updates are only through appends. So there are ways to go back and update some of the data, but it's a little more complicated to do that. So the index file is just a binary file
10:25
containing offset and size. So when you do a lookup for a tile, it works really fast. You just look for the role. So at the top pyramid, you have 00, then 01, 02, and so forth. You scroll down the road, look for the offset, and then read for that particular size of the tile,
10:45
and you get back that imagery really, really fast. So moving on to the actual, what I call the on-air server, it's just an Apache module. So basically, you can run this software with
11:02
any Apache server. It's just a plugin that you can configure. It contains the cache metadata. It's basically the information we saw in that MRF header that's stored on the server, and we were able to read information about that layer really quick. For cases where there's
11:25
no data, so much the Earth, if you only have a layer that's just the oceans or just the land, you might have some empty data. So we have the special index 00 that just returns a blank empty tile. I mentioned earlier about the common protocols. We have the OGC-WMTS,
11:43
Tylenum SNKML. If you're familiar with any of those, there's Git capabilities that a client application can read and figure out what layers are available and so forth. This isn't actually done through the Apache module. We have a separate CGI script that will return you the Git capabilities information.
12:04
We don't exactly follow the specification for WNTS. We have this added time extension. Much of our data has historical imagery, so we have to go back in time. So we had this little time snippet, if you will, to be able to create back for days in the past.
12:29
With KML, just a little pointer, you always get back a global imagery. So when you pull a KML file, it's just for the entire Earth. Whereas with WMTS or Tylenum SNKML, you can get the individual
12:43
tiles. The whole purpose of the module is really just to translate HTTP requests and be able to read MRS and return that to your client application. So a little bit of information about
13:03
performance. This is sampled from MODIS data. I guess the key thing to get here is it's possible to get speeds of over a thousand tiles per second. This is without network latency. So these numbers are a little bit dated. These are from 2011, back when we were doing
13:25
an evaluation for Gibbs. I'm not sure how this compares with other tile servers nowadays. It'll be nice if someone could maybe check it out, give an evaluation, and there's better technology out there. I'd love to know about it as well. So yeah, I encourage just some comparison,
13:46
see what you guys think. And just to see how on Earth is working out in the wild, these are some sample metrics from Gibbs. There's about 40 million requests two months ago,
14:01
so there's a pretty decent size of visitors hitting the server, and it's not really putting any dent into it, so that's a good thing. Currently there's over 90 image products, and with Gibbs we only have 250 meter data, but in the past we've worked with Landsat 30 meter, 50 meter data before. So to help
14:32
configure this server, we have a couple other tools just to make life a little easier if you're using on Earth. So there's this MRF generator. You can use GDAL to
14:44
basically generate all your MRF files, but this MRF generator abstracts all that GDAL command stuff away from you if you're kind of shy about using that stuff. It also allows for automated processes. We use XML configuration files, so another system could kick down some XML files
15:04
and be able to generate the imagery for you. So there's also this on Earth FLIR configurator. We use that to basically generate the server configurations, the metadata for get capabilities and so forth. And a couple other tools, there's this on Earth legend generator.
15:25
You can take your own color map. We have our own Gibbs format for color maps and just be able to generate a nice little color graphic using that tool. And there's also this metrics tool which basically it just converts Apache logs into a
15:42
format that's useful for if you're acquiring metrics. And this graphic here kind of summarizes how the system works. You prepare the imagery. This assumes you have the images available or global mosaic and you're able to turn that into an image pyramid. And once that occurs,
16:03
there's some configuration involved with the product layer. And you load that into the server and from that point on you can serve out the imagery using these common protocols.
16:21
So for Gibbs we use on Earth in conjunction with the software called Tye. It's an image management and workflow system. It helps automate and manage our MRF generation. Another really great thing about it is that it keeps track of the metadata and this is good for data provenance which is important for scientists to be able to go back and track where
16:44
the data has come from. Another great thing about Tye is that we can use it for searching and and be able to search on the metadata and so forth. This is also in development at JPL. It's not open source but something good to know about. So I was going to give a little preview of some of
17:10
the client applications. Looks like I have some time to maybe check these out. I wasn't sure about the network speeds here. I had a little bit of trouble but let me see if...
17:29
So this is world view. We use this as the Gibbs reference client. And what on Earth is really doing is serving the detailed imagery here. If I hit refresh
17:44
you can actually see the imagery or the tiles being loaded. But like I mentioned before I think there are some network speed issues. So I'm not going to do a full demo. There's another
18:01
presentation coming up at 4 pm exposing NASA's Earth observations I believe it's called. And there's going to be more in-depth talk about world view. So you can actually see that tile showing up here. And that's basically what on Earth is doing. There's a couple more other
18:21
client applications. This one's using Google Earth. You can't really see it on screen here. Just do this screen resolution. But then this is showing sea surface temperature. And there's also a 2D version. This I believe is using Leaflet. And it's showing the same thing. Sea surface temperature. There's other satellite layers that
18:45
you can look up and so forth. One thing I forgot to mention earlier. This on Earth survey only serves up raster imagery. That doesn't stop you from using a separate server to serve up vector layers. I believe this has some vector information
19:07
somewhere. We also have this lunar mapping modeling portal.
19:24
It's taking on Earth to the moon I guess. So we have then this is pretty cool. You see just imagery back to the Apollo era. historical archive of imagery of the moon. So just pretty cool.
19:49
So those are just a few couple of sample client applications. Here's some links here if you want to check it out on your own free time. See what they're all about.
20:07
So here's a list of other known support clients. Showed off world view which uses open layers. Here's Leaflet, Google Maps, Spring Maps. And of course this is also available for GIS clients. Esri ArcGIS Online actually has the Gibbs layers available
20:27
through their portal. So you could use that for a variety of GIS applications. And of course there's mobile clients and script level access which is useful for if you
20:43
want to suck down a bunch of tiles and build a cloud-free map or something like that. I believe Mapbox did that for their own version of cloud-free maps. So a little bit about our
21:01
open source project on GitHub. All of this code is available online. There's a couple other projects as well. There's world view and some samples of how you can build your own clients as well. The primary source for the On Earth software is on this NASA-Gibbs On Earth site. We're also working to separate the MRF code so we could build that into its own standard.
21:27
There's some specifications. It's not well documented at this point but there are some specifications up there to provide you with a little bit more information. Most of the code is in C or C++. Some of the non-performance-centric software we use Python
21:45
just because it's easier to develop and use. And it would be nice for the open source community to check out the software, provide some feedback whether it's good,
22:00
if you like the performance or if there's some other software that maybe could be a benefit. And also if you participate, there's also the sense of helping out the Earth science community and providing some aid to NASA and pushing technology forward.
22:22
A little bit of a preview on the future developments. We're working on granule imagery. The challenge is how to be able to serve granules via the On Earth server without sacrificing any performance. There's a couple other items such as multi-day imagery
22:43
and several improvements in the MRF specifications. We're getting some help from Esri. They're cloud-friendly, I guess. That's basically the end of the slide, anyway.
23:12
I did want to advertise there's an open position. I don't know too much about it. Yeah, yeah, pretty secretive. But yeah, that's basically it.
23:37
So I'll leave off with a couple links here. There's just links to our GitHub page.
23:51
How to use Gibbs. Basically there's an API. If you want to contact anyone at Gibbs, there's the email address and my personal email address as well up there. So I guess at this point I can
24:04
take questions. I assume that the serving is all done from those pre-generated tiles? Like
24:20
when you talk about the color map, that's not applied on the fly. That's applied during generation of the tiles? Yeah, yeah, that's correct. All the tiles are pre-generated and during that pre-generation process, we're passing the color map and that sticks with the imagery. So for instance, if you just generate JPEGs, then that's the only thing the server can serve. Is there any interest in doing dynamic conversion? I assume
24:41
that kind of defeats some of the performance purpose. Yeah, there is interest but that's the problem is performance. Might be slightly unrelated. Is there a feed of the different data currencies of the different products within the viewer? In other words, how do you best monitor
25:01
the freshness of the different data that's available? I'm not sure if I followed that question. The imagery tiles, you know, the recency, how can you best evaluate the recency of the different tiles in the imagery? So for Gibbs Anyway, we serve up daily imagery.
25:24
So it's basically just time-stamped into that data. There's not really a way to, if you're looking at sub-daily imagery, to see how, there's not a time stamp for how recent that imagery is.
25:50
Are you guys just showing the EO bands or is this any other of the derived products like the sea ice concentration cloud cover analysis, etc. Is that going to go into
26:01
to Gibbs, any of that sort of thing? Or is that the other topic? Yeah, there's, so for Gibbs there is, if I go to world view, there's several layers of science products on here. So in this one, you know, oops, you actually, you guys don't see it. Let me try to get here.
26:27
So yeah, there's different size parameters that you can get. The list of
26:53
compatible clients is quite large. Is that because of the WMS support or do you have, you also support specific other protocols? It's mainly because of the WMS and WMTS, which is
27:07
commonly used, or KML. So if, in theory, any client that can, that follows those protocols can deal with that, can access the data. You mentioned that the WMTS had a little extra
27:30
thing for time. Is that above and beyond sort of the WMTS standard? And then does the second up question is, does the WMS support the time coordinate or the time parameters?
27:44
So WMTS is something we added to, it's not in the WMTS spec, it's something we added. And I think, I'm not a WMS expert, but I think there is a time extension. And so that just follows along. Right, right, right. Oh yeah, yeah. I was just wondering, with the
28:18
MRF format, if imagery is updated, you know, four hours from live, and the entire format must
28:25
be appended for updates, how does that happen and how have you dealt with that? Is the entire, does the entire file need to be regenerated when the data is updated? Yeah, pretty much at this point. About how long does it take to generate the MRF file?
28:50
That totally depends on the size of the imagery. If it's kilometer size, it's pretty quick. But if you're dealing with, say, 3 million Landsat data, that could take hours.
29:03
So that's, there is some limitation there if you're dealing with really high resolution imagery, it's going to take a long time, unless you have some really beefy computer power to handle that.
29:23
I think all the questions are done.