Monitoring Ozone Levels with Postgresql

Video thumbnail (Frame 0) Video thumbnail (Frame 1231) Video thumbnail (Frame 2244) Video thumbnail (Frame 3209) Video thumbnail (Frame 4366) Video thumbnail (Frame 5602) Video thumbnail (Frame 8086) Video thumbnail (Frame 13869) Video thumbnail (Frame 15388) Video thumbnail (Frame 17865) Video thumbnail (Frame 20506) Video thumbnail (Frame 21918) Video thumbnail (Frame 22806) Video thumbnail (Frame 23868) Video thumbnail (Frame 24830) Video thumbnail (Frame 26669) Video thumbnail (Frame 27656) Video thumbnail (Frame 29248) Video thumbnail (Frame 30133) Video thumbnail (Frame 31147) Video thumbnail (Frame 32385) Video thumbnail (Frame 34765) Video thumbnail (Frame 38453) Video thumbnail (Frame 39402) Video thumbnail (Frame 40602) Video thumbnail (Frame 43415) Video thumbnail (Frame 47652) Video thumbnail (Frame 48813) Video thumbnail (Frame 51340) Video thumbnail (Frame 54227)
Video in TIB AV-Portal: Monitoring Ozone Levels with Postgresql

Formal Metadata

Monitoring Ozone Levels with Postgresql
Title of Series
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Monitoring Ozone Levels with Postgresql Database Streaming Replication and Monitoring Postgres is used to manage data from the Ozone Monitoring Instrument aboard NASA's Aura spacecraft. The database implementation must handle large volumes of complex data transmitted continually from the satellite and generated by processing-intensive analyses performed by a team of atmospheric scientists. This talk will describe the architecture and some of the challenges faced. Focus will be given to our replication efforts, software developed for monitoring, and ongoing work to create a decentralized network of services commnicating through a RESTful interface. NASA and its international partners operate several Earth observing satellites that closely follow one after another along the same orbital track. This coordinated group of satellites, is called the Afternoon Constellation, or "A-Train", for short. Four satellites currently fly in the A-Train: Aqua, CloudSat, CALIPSO, and Aura. Each satellite has one or more observational instruments that are used together in the construction of high-definition three-dimensional images of the Earth's atmosphere and to monitor changes over time. Aura's instruments include the Ozone Monitoring Instrument (OMI). Data management and processing services for data harvested by OMI are provided by the OMI Science Support Team headquartered at Goddard Space Flight Center. Raw OMI data is received and initially processed at a ground station in Finland, then ingested into the system, where it is analyzed by scientists who submit processing jobs. Earth Science Data Types (ESDTs) are the products of these jobs, and one of the principal types of data managed in the database. Complex and abstract, ESDTs represent the interface between the raw science data and the data management system, and more than 900 are currently defined. Our current database implementation includes 10 clusters, each running Postgres 9.0.4, and divided into three production levels: development, testing, and operations. The central operations cluster handles on average about 200 commit statements per second, contains tables as large as 160 million rows, and is configured for streaming replication. New data is continually being added to the system, and the total quantitiy is increasing at a rate of about 60% per year. This influx of data, in addition to scientific analyses, can cause the load on the database to vary suddenly, and monitoring software has been developed to provide early warning of potential problems. The latest implementation of our software architecture uses decentralized services communicating through a RESTful interface. Databases are bundled together with their software component, and schema changes are managed using patch files. A utility has been created to apply the patches, and ensure schema consistency as the databases are amended. Perl's Rose-DB is used as an object-relational mapper, and database queries, via HTTP requests, are supported by encoding the query information into JSON. The new platform uses a different data model, making it necessary to sync between the two representations, and causing some difficulty with data duplication.
Process (computing) Computer animation Sequel INTEGRAL Software developer Energy level Bit System call Computing platform Row (database) Physical system
Estimator Computer animation File format State of matter Network topology Surface Multiplication sign Energy level Nichtlineares Gleichungssystem Wave packet Power (physics)
Multiplication sign Planning Survival analysis Bit Mereology Regular graph Measurement Data management Computer animation Database Estimation Whiteboard Quicksort Data type Pole (complex analysis) Set theory Physical system
Standard error Process (computing) Computer animation Multiplication sign Moment (mathematics) Bit Resultant
Trail Group action Server (computing) Multiplication sign Execution unit Motion capture Online help Function (mathematics) Metadata Frequency Operator (mathematics) Database Queue (abstract data type) Cuboid Energy level Diagram Physical system User interface Electronic data processing Process (computing) Mapping Cellular automaton Software developer Bit Volume (thermodynamics) Cartesian coordinate system Numerical analysis Subject indexing Arithmetic mean Process (computing) Computer animation Logic Hybrid computer Phase transition output Right angle Family
Slide rule Computer animation File format Different (Kate Ryan album) Database Metadata
Casting (performing arts) Sign (mathematics) Computer animation Strategy game Mapping Multiplication sign Single-precision floating-point format
Area Ocean current Information Multiplication sign Weight Shared memory Coordinate system Insertion loss Maxima and minima Demoscene Dimensional analysis Mathematical morphology Frequency Crash (computing) Computer animation Software Personal digital assistant Asymmetry Family Operating system
Goodness of fit Information Query language State of matter Deadlock Personal digital assistant Physical law File archiver Pattern language Table (information) Family Power (physics)
Web page Area Server (computing) Computer animation File archiver output
Server (computing) Graph (mathematics) Scaling (geometry) Computer animation Average Software developer Operator (mathematics) Right angle Price index Event horizon Stability theory Physical system
Computer animation Synchronization Square number
Query language Different (Kate Ryan album) Square number Numbering scheme Quicksort Set theory Physical system
Backup Computer animation Information Computer file Personal digital assistant Business model Bus (computing) Configuration space Real-time operating system Parameter (computer programming) Computer Computer worm
Standard deviation Decision theory Software developer Parameter (computer programming) Representational state transfer Cartesian coordinate system Measurement Web application Component-based software engineering Computer animation Database Order (biology) Energy level Right angle Computing platform
Web page Modul <Software> Greatest element Game controller Server (computing) State of matter Patch (Unix) Execution unit Numbering scheme Insertion loss Mereology Web 2.0 Database Operator (mathematics) Business model Software framework Physical system Standard deviation Programming paradigm Mapping Interactive television Sound effect Volume (thermodynamics) Representational state transfer Web application Computer animation Personal digital assistant Statement (computer science) Software architecture Quicksort Table (information) Asynchronous Transfer Mode
Server (computing) Presentation of a group Computer file Mapping Multiplication sign Cartesian coordinate system Element (mathematics) Web 2.0 Computer animation Operator (mathematics) Database Energy level Software framework Object (grammar) Software architecture
Trail Server (computing) Computer file INTEGRAL State of matter Patch (Unix) Numbering scheme Online help Mereology Mathematics Mechanism design Component-based software engineering Database Software testing Set theory Addition Process (computing) Software developer Bit Special unitary group Instance (computer science) Surface of revolution Web application Computer animation Software Repository (publishing) Quicksort Procedural programming Table (information) Spacetime
Building Musical ensemble Computer file Execution unit Web browser Mereology Event horizon Computer icon Product (business) Operator (mathematics) Database Business model Software framework Object-relational mapping Plug-in (computing) Addition Touchscreen Software developer Feedback Numerical analysis Residual (numerical analysis) Data management Wind tunnel Loop (music) Computer animation Software Personal digital assistant Pattern language Quicksort Table (information) Marginal distribution
Modul <Software> Standard deviation Process (computing) Open source Hazard (2005 film) File format Multiplication sign Workstation <Musikinstrument> Projective plane Tape drive Web application Word Computer animation Software Personal digital assistant Repository (publishing) Database Business model Metric system Reading (process)
Computer program Group action State of matter Multiplication sign Disk read-and-write head Web service Identity management Social class Physical system Gradient Data storage device Bit Representational state transfer Type theory Exterior algebra Order (biology) MiniDisc Right angle Quicksort Whiteboard Server (computing) Implementation Game controller Service (economics) Computer file Wage labour Machine vision Metadata Hypothesis Average Database Computer hardware Data structure Stochastic kernel estimation Address space Set theory Condition number Validity (statistics) Content (media) Division (mathematics) Directory service Cartesian coordinate system Uniform resource locator Computer animation Integrated development environment Friction Software File archiver Point cloud
and this is actually here in the front row axis the dubious and on the other developer and assisted EPA although only science supported at Goddard Space Flight Center and we're going to talk about monitoring ozone levels with PostScript sequel so
I'm going to begin by giving just a little bit of background and outlets will talk about how data is brought into the system during the process call ingestion as integral over some of the challenges that we face and then I'll describe some on going developments in the software platform that affects how we manage the data so what
about 450 miles above the Earth's surface is a formation of satellites that are unipolar world that that takes some across the equator of 14 times each day and they call that formation a tree and I have been members in a trainer or not observing satellite so they got their instruments when it downwards instead of power that could be seen as a train that you probably just barely see here is a satellite called on matched in July
2004 also has 4 different instruments for of monitoring gross atmosphere of the use monitoring instrument is a instrument for monitoring stratospheric ozone levels and that's our estimate 1 provides a state of paralysis because these guys here
and I have a plan there whether that they go over the north pole each time they go around the globe of the data is received a 4 1 all on ground station part of like this 1 and also board Norway all of the data is
processed a little bit at the ground station and converted it into something called an Earth science data types or EST t this is not really the sort of data type that you would normally encounter in a database and it's really more like a procedure for of processing a set of measurements and defined by scientists which of matter of scientists of but they provide survival the well enough it's between the data coming in from the satellite and or data management system and regular there's more than 100 that have been defined that we have to store it in the data so
that all of these the ICT is argued process standard error generated and the ultimate goal is the atmosphere scientists will use them to use of the results of all this processings to create an atmosphere that trials and climate change balls so that is not
going to tell you a little bit about the process of how the data gets said in the past and processes morning in moment OK to time he was them
I want to show this diagram to have a higher level of that the system on a processing system that is seeing and a box that you see that diplomatic their periods slice showing that the ground system capture all the data coming down and will be used send the raw data to allow to us that all or they do some processing certain level data and those of the light of the data and they will get into our input to our our data processing system and that's not all we have a lot of large group the scientists community and rely on all help output data queue before he application and doing research that those cells scientists will will going from same using go for the mapping of phase to making requests and that they get to the for to data received an all all only Pajek operator also to go through a lot can go for our web interface to 1 monitoring of processing and also some beating job through the planner and see what kind of data the 2 need to be the same time we need to be processed data for any new public communal or some of sand is changing the logic to how women and manipulated the input data and come up with the final product and for for them but in the face
and this diagram is sewing and as soon as that of the data basically have to wait to store the data in the real data will all unit database of light you sure that all the usual and capture data will be standing systems them so that they data is a very high volume and the manic hybrid data we will all have to process and this was staying Metadata Server enough that's higher than the PostScript but I can capture tracking although the metadata final system on that two-way go out will trigger triggered but he just process of the true them all you just which is reliant on the right out the new data and we have a logic to checking of human checking any any of the incoming file so that you know are kind of and then that will trigger you just brothers and based on certain criteria was set to during the process of those data and that's terrible when the operator also can have some meaning in a job and contracting the just we can already had and seeing a planner and see low and also will be able to also something job that way and those that not number I need to explain a little bit more about our ongoing situation and we're using tragedy almost all over 10 years the tracking or the metadata and you know that it's a system of bagging a few years ago we realize that that is the that the demanded to really do so we are standing on the outstanding another new view on the server which you see in the know down in the corner of the corner there and the axes of new systems ongoing development system and basically on that system already partially in operation some application already they're planning to to that and that the idea is we need to have of the metadata update of 1 best system but then the basically that the popular priority is make sure that legacy system is ongoing as without using any data so we're based on that once we updating metadata and legacy metadata and then we will another trigger will go into the and the index of Metadata Server on how accurately we do from time to time with you have submitted bit pogroms send either and the reason that we did in the 1st set so you have 100 per cent all the data going into andX metadata so that when you see the box along the synchronized data the process we trying to catching and amusing any if any of the problem of bad happened successfully in judging both is to do
yet how can you can you repeat
again they of the are now about to databases that used to different data that is not the same they have the same yet on campus so you more in the following slide on that said well 1 of the challenge we are facing the you know trying to find what they think that that's basically the same kind of metadata about in different format of the schema is totally different we'll have different conditional all this have have different forms of the so you
how it in this thing that I wanted to have a baby sign desynchronization this 1 of them are backed up by a single nice cell replication with red with this strategy we are using but I want to talk about the history of before we running on an island we was running a and we still have been no replication that time we had pretty map based on based on the PG done to back up I've seen that they they're getting very big and the time to doing a them and also and that the performance will drop while doing that that time we only have 1 server and and also asked that the dam and the data on going not there and the between the 1st and then the 2nd them that we know we are pretty magic that that the better they crashed and we're using those and the eradication I a really happens to solving this problem we have is we have of standby databases and move it better than but that was still continue to do the dumb but the last 3 treatments from that so we can do it from the slave and everyone casting of of performing the master I that's a
dimensions some some of the same asymmetry that the thing we we have focused on on how to make sure the replication successfully with of this of this so the current topic area the possibility of the the way that have applied properly and then can can cost the replication fail so from between their commonsense less some kind the network is unavailable for a period of time we have to work and notice of all that and so on and so has ability I would say maybe make the leading some morphology the world and you see that this side we have a PC that are going on doses a copy it can take a along longer period of time and finished what the replication going on scene we we're coming from the slave and then have some coordination need to be done I'm going to show you some cognitive we adjust to avoid the fatal complete the the PG done while the replication going on actually just parameter sharing is a maximum the maximum loss then Vice Chairman ideally we set it to 1 to make sure on replication during the way to fringe that avoiding the crash crashing and you can see where we
have some family to you based on the PGA up here that's the operating system to showing that the status on the house and 1 the replicating going on we we're kind of monitoring those that that you get through to a long period of weight then we notice that something is going on and perform and they have a lot of the problem so we're trying to trying to have some to automated tools based on both information to alert the a and via and we'll also using the log and capture some the information then case we be to to publish it in the period of time the performance of ship and that's now
you you know that in close good money tables is invaluable you have both information and also we can use those creator and the creator how does the law applied and again I'll move on certain pero pattern we also get a lot and things like that and that's where powerful query tables is incredible will help us to monitor and there was some
kind of but they will also how about that make sure I that family is seen archive in case we need to rebuild or restore and also and make sure we don't capturing the wet-lab before actually this lady applied in and out and we would attract attention those those displaced and also the monitored away state how how long this thing and and also monitor any deadlock situation happens and that's because
you know we've been using to alert and we haven't had any pages in the Tribunal's work 24 hours so we don't need from in that and because of that because I was like getting a lot of data and input data we always know archive server the western area we can always go back and also data can be with you would use
this is just a from some looked about the performance you how much how much the data
we have and I Granny will where we have about 10 questions about the server we in different stage of the development we interact we have development server server and we had pasta we have operations server and on average and they can be up to 100 commit per 2nd some heat up to 10 thousand if you kind of work will this average on natural and like I can think is really difficult to match shared that have that they don't have a really the standard way we would measure the habit of the event and this is how I know some some indicator how how big that I just wanna show you 1 of the biggest stable without 160 million role and that of the graph right about 40 60 % the last 3 years and I
just have some charting of this a legacy system have 2012 really going nizing as may be expecting more of something that is very difficult to be predicted well and especially understanding we have may have a lot of unlabeled data John they that anyone of reprocessing 10 year of data they can suddenly and global data we have so this this a set that that trying to scale system and see we can handle that come along and this is new
system but that doesn't take a
chance of being mentioned above synchronize announcing the new CEO of this right cumbersome and also that because they going up they get data from make it they difficult to not and use 60 million rural data which 1 is the new 1 biology we and also which 1 is different when we're trying to synchronize the square that and trying to identify what I I think best habitat to to synchronize 1522 thing analyzes is easier than status was still working on trying to find a better way maybe I trying to see any any audience can deal suggests a better way you have different schema how to do this synchronization that you see
this this and that is legacies scheme of the weapons news tho for any detail but this this show you the new 1 will be
simplified but basically sufficient data in different people and things like that and that 1 when and that desynchronization square this pretty much of the mapping the 2 different schema and and trying to sort of standing right query to to to update of missing data in the new system is a set of candidate
had just want to show you that this was a snap the computers of the the configuration of the transcript config file as a way of putting a payload where we by sigh easier to identify it that doesn't may be helpful for beginners never use of replication may be helpful in that there's certain parameters you know on master and slave is there are different and have a you can read that this anniversary homework and you can't have a lot of the people can have a lot of very detailed information
and that has to the exactly before that that that we have to make adjustment for because of the heat doesn't we put minus that of what actually happened at the bus stop landed them during the damn will not be doing in real time but just so it doesn't matter what style cases it doesn't need to be 100 % real-time you know slave for using a slight pretty metaphor for for for the backup purposes and also some who we only really really creosote actually doesn't need to update we eating our of slave to offloading the of the the model from the last thing that happens through the what are the performance of the master and this is some
other permanent parameter we use that's it for me and say
that this morning the amount of
developers has been working on the rewrite of our of processing platform and of some decisions were made in every right that affect the way that we are all going to manage the data from here on out and so I wanted to go over some of those things where the the budget that's and and the new platform is that we're doing a lot of decentralized kind of components so that instead of having a more monolithic design we broken all out and these are components like component x y and z here and what was appropriate we also splendid a little piece the database of an integrated that with its component application and then these discuss communicate with each other yeah using rest so and I I measure of everyone here knows level was a question of what to expect in the audience is our 1st PG con 7 100 people on developers but rats is you know basically using standard HTTP request in order to and interact with a web application and use
a free part so 3 things that I think are interesting to mention on how this new architecture affects the vitamins and what is considers the web framework of 1 concerns the warm that's integrated in and out of the other is the system of database patches that we've implemented for doing scheme of things so all over each 1 of these 3
on the ground framework is of course just something that that helps to facilitate creating a web application in our case for using pearls modulations framework and modulation implements a model view controller of paradigm where of course the view is what the user sees in this case is a web page of the model is a state of the system on all of the database of most often and the controller is kind of a mapping in between their interactions that user of since in through the year to do something on the model and here at the bottom and you can see here's an example mappings of volume was against this is rest so we're using and you have also for the users to do the interaction the sitting on some sort of HTTP requests and that actually has an effect on the database and so in this example we've got I get request which maps to select statement on a person table and we have a lot of post requests which would map to of an update on insert statement on on the same table and Italy which we just want to delete on and but that Apple standard I guess for web framework the thing that's a little interesting about about our system is we've extended mode of that so that when this the web server starts up it automatically of recognizes the database that it's coupled with so it's in it'll and that it has an associated database check the database rather tables and it cannot generate all of these units that are out there are use by a web application to do the the basic crowd operations you don't have to manually define all these things are of the city the
never architecture also has an integrated all elements the and the quizzes object relational mapper ensure everybody here knows that and similar to the web framework we've done some things to help automate of operations of the 1 that has so as the server stops all of the all I'm kicks also reads the configured file and will try to discover that the the database as well and the other 1 will generate the whole object methods needed for those of present operations center we would basically at 2 levels
bone of database operations mappings going on here so that the web server aspects and the web framework is going to generate a are of allowed without a lot of your time out so that a user would use or the application the user is using is going to use and these then make use of the object methods generated by the warm which can then go about and manipulating the data on the back and so a lot of this happens just magically but for when when create a new application with this framework and you define your database of these methods and that's just get instantiated for you know that I
have but I would say 1 thing that can be an expensive so if the database is the size of all of you it's going through all these tables generating of all the operates and so that would be needed to manipulate them but luckily since this is a web application and that's when the server starts absolutely pay the price went on and so forth has intuitive problem so have a sense we're doing sort of distributed development lots of little components that can be developed individually but it was desirable to have a way that developers could not create schema changes as the working on the component because remember these components of these web applications have that is tightly integrated in their their own homes vesicle database and so what we did was in addition to the current in the get repository for the component we added that have this scheme definition file in this space that's equal parts and then as you make is of this changes were made to the schema of the database but developers and patch files are very much in the same way that you might have had a patch to suffer you added of DDL patch here and so on new patches come in and there are other committed to the repository eventually you want to of my great these things out to a running instance of components that's all that all more exactly you want migrate and to the the database that's running out within component of X and the software is going to do a little bit of help in
and to maintaining the interesting integrity on these patches to make sure things to get all decrease in but the besides will make of sort of so let's imagine that you have a new patch that comes in here so they that the battered patches and as a patch is added to the get repository and tested during the the the the process of testing and the Committee to the repository this scanner definition file is rebuilt is regenerated to patches that have come out later and an entry is made in this text file called patches applied . txt take as sort of a bookkeeping method to keep track of all the patches of and gone into based obstacle that when we're ready to apply new patches over 2 of a running instance there's an analogous table you can hardly see it here but it's patches applied just like the the text file in the get repository but this is a regular rests table that's keeping track of all the patches that have been applied to this instance of that of the sun so all we're doing that the the patch software will do for us is take the schema of the running database check and see which patches on your competing compared patches applied that text file and the patches applied tables see which new patches would would need to be added to this database and it can do a day of the states a hypothetical that that that you would get if you took the existing scheme applied all these new patterns and if you compare that to base that's equal the Schengen the same thing so assuming nobody's done any owner-operators changes to that so it's a way of checking to make sure nobody's been looking revolution of the should go through the the patch mechanisms and do it the way we decided to do this and everything like set of of the the yeah so that that procedure would be something like this you you will have you check out you get repository and all the patches have also been there development previously committed but this past was built on top
of bonds margin building is an extension of this model and model that there is a lot of software for managing of the installation models normally and in this case we just extended that to manage the the installation of the patterns as a set answer your question so that me and in addition the user model building were using residue as the object relational mapper and as I said no delicious is a rather new framework that has that fairly popular in the last year OK
and I have also been working with our hands on some sort of monitoring tool for all our databases and as he said we've got around 910 clusters we there's some that possible clusters that we have that we were really concerned with something happens so that we've got a number of others and what would like is kind of a simple tool that provides a something like a weather forecast of of how the the clusters are doing and so I created a little while that of using 1 delicious this of been doing this development were already on that that where the reader configured file of but at the adjacent formatted configured file and then and then for this further step that was on a screen upon in a in a browser actually and it just gives an icon for each of 1 of the clusters that are color-coded to give an idea of how this thing is doing and it's just a product out right now but I without reading it in the ground and thrown the set up on the screen and it starts JavaScript event loop that have was are some of the crowd that create the SSH Tunnels out to these these databases and then start doing various monitoring operations and then that's the part where I want to have some feedback from people because I'm not sure exactly what's the the best things to be checking on others units commands you can check some step tables and if you some of these bands we had the younger then you may want to click on it and
this case a quick agreement and that would give you all of not formation again this is that the uh so it's not just 1 standard web application these web applications you create with modulus is a very light see could just check him out of your repository and you can start on your workstation in 5 minutes you know the better the essays will read access anything and that's what we want is something simple to to monitor about this so any suggestions of like and I guess that princess
the questions this I don't have it hazard anywhere that it turned out that and I'll be glad to just show that this isn't that kind of alone time to observe a similar step we couldn't get up like that but not develop that stuff is kind of company stock but the metric tools for model project and is that the kind of cluster Rishis because we we have clusters of databases that that of process things and this is the guy who started module or actually what is in the past and who are the Hispanic cluster trying to release some of the software and we just haven't devilishly hard time of it because we were actually contractors were working for the government and the just the amount of red tape and trying to open source and said that were redolent of abandoned and what we can release to in things that I use the word of the head of that the the question about then we can go back and assets may
wanna just that I think he
mentioned this back in and you
use it to use women on the on the long and you will only get 1 this of year following only know you and you will go on and on and on and on and on you know what you need to do it all the only way you when you go on and on and on and on and on and on and on and on and on what you see on the board of the on the on the phone from the but if you really go on and on and on and on and so on and on and on and on and on and on and on and on and on and will what we have another this established and this is usually a yeah I just said this is what happens when I and this in the not that I'm aware another is a set that we're out for that's it that's it uses a relational database and that what it does so we don't know what as he said there's millions of files that costly coming in from the satellite as if it never rest and how we didn't and the fat some on those and other store them in a in a distributed fast of valid archive server that can span across multiple servers but the location is determined by the in the 5 so on of all the contents of the file so a lot of you you you can easily just generate the address on the fly like that that you are a segment but it's it's not going to use a lot of Unix type stuff and just to the in the fat some stored in directory that ended and the directory structures based upon the fact that in the fabric of that just so as to it's generated by that what what you get that from that so it's not too complicated really other programmers because there's so many of the she said in and sometimes you have to rebalance the throughout the service of that so that 1 server doesn't have to mean of that haven't had this and then they said before at 1 time we were of storing the locations of the database but always happen injustice files 1 of the things of said that the the American press mostly would work with the atmospheric scientists to etc. process that after considering in the hands of the the users other from the front and back and the hypothesis is this is the kernel for all kinds of and in your government funding and is formed from and how do you know what you think of the view that knowledge the identity of the holder the right averages certainty you know long and easier to apply it to the gradient and actually not only will not file that you will you know how food and also the data and things like that and you all that matters you get a lot of the middle colonies were pretty much the reality is to that of the state of the environment conditions on the goal of the of the of the of the vision that all of the of the order of the of of the group where would you like some what about you know that the services of exactly and the cost of housing and it having exactly that that I did that for some clouds that is the heart and soul and I know there's a kind of division of labor here that's that's critical friction recently so I have the that team without the set where the stuff the desert the facts and stories of files that have these assessed essays over at NASA managing the hardware back there and they can implement that any kind of red where whatever just system they want and I recently there was a problem of like files were were going and that's enough there was some the letters and there was a little bit of finger-pointing well you know I didn't do this to you guys wrote this crazy software and that but everything that you look at its centre you you tend think that it was something wrong with that of the disk of the that out but I I don't know much about the way the hardware implementation of and it's you know we could change without are control or presence of I don't I not many terabytes of your notes that willisi of from 2004 to 2 now for only uh so there's been about 8 years worth of data in a manner that want you know the guy down the classes with who wrote this this solution I I think he invented the company to write himself but we we already had so that implies that it was it was a quite right and that that body agrees that we needed right of put ourselves but we didn't and there was some clever stuff you put into it so as I mentioned that this is cause a little bit of friction a start to and application of replication I have yeah yeah that you do not know how to do that we do not have to be in the form of a woman who was acting as data on the complement of the act as an alternative to the quality of the people in the but I think you are very much like that sort of thing that you're types the metadata so there's a lot of 1 of the most common of 1 1 1 1 1 then we can will have a lot of people want to what is actually 1 of the our on the other side of the head of the I mean by but they say