Monitoring Ozone Levels with Postgresql
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 20 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/19027 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
PGCon 20128 / 20
1
3
6
7
9
11
12
14
15
16
19
20
00:00
Row (database)Thermodynamischer ProzessSequelINTEGRALSoftware developerSystem callBitPhysical systemEnergy levelComputing platformPersonal digital assistantXMLUMLComputer animation
00:51
File formatNichtlineares GleichungssystemGroup actionSurfaceMultiplication signNetwork topologyWave packetPower (physics)
01:30
Energy levelPhysical systemState of matterEstimatorPole (complex analysis)Multiplication signGroup actionPlanningMereologyWhiteboard
02:08
QuicksortInterface (computing)BitDatabaseProcedural programmingData storage deviceSet theoryData managementMeasurementData typeEstimationSurvival analysisPhysical systemRegular graph
02:55
Standard errorThermodynamischer ProzessResultantProfil (magazine)Energy levelPhysical systemBitDiagramMultiplication signMoment (mathematics)DiagramJSONComputer animation
03:44
Numerical analysisProcess (computing)Thermodynamischer ProzessBitVolume (thermodynamics)Electronic data processingMultiplication signFunction (mathematics)Hybrid computerTrailPhysical systemDiagramCuboidQueue (abstract data type)Operator (mathematics)Server (computing)MetadataCellular automatonMappingRight angleCartesian coordinate systemFrequencyMotion captureFamilyLogicEnergy levelGroup actionPhase transitionUser interfaceoutputArithmetic meanOnline helpExecution unitSoftware developerSubject indexingDatabaseProjective planeProduct (business)Type theorySlide ruleDemonInternet service providerComputer fileFile archiverSet theorySynchronizationMedical imagingDiagramProgram flowchart
09:15
Different (Kate Ryan album)Table (information)PreprocessorDatabasePartition (number theory)File formatSlide ruleMetadataComputer animationDiagramProgram flowchart
10:16
Stability theoryServer (computing)Multiplication signReplication (computing)Graph (mathematics)Square numberOperating systemPower (physics)Table (information)Maxima and minimaSynchronizationDimensional analysisInsertion lossNumbering schemeInformationShared memoryPhysical systemLevel (video gaming)FamilyState of matterSoftware developerCrash (computing)DeadlockFile archiverAreaCASE <Informatik>AsymmetryRight angleQuery languageOperator (mathematics)Sign (mathematics)Web pageWeightSingle-precision floating-point formatoutputOcean currentFrequencyMathematical morphologyScaling (geometry)AverageStrategy gameSoftwareEvent horizonGoodness of fitPrice indexCasting (performing arts)Physical lawCoordinate systemPattern languageDemosceneSoftware testingCore dumpComputer fileDifferent (Kate Ryan album)EmailWorkloadElectronic visual displayMereologyDatabaseSlide ruleStructural loadRow (database)Context awarenessBackupBuildingParameter (computer programming)Bit rateOnline helpScripting languageInternet service providerStreamlines, streaklines, and pathlinesComputer animation
19:39
Physical systemSquare numberDifferent (Kate Ryan album)QuicksortSet theoryQuery languageSynchronizationScripting languageTable (information)Replication (computing)InformationSelectivity (electronic)Real-time operating systemParameter (computer programming)BootingConfiguration spaceCASE <Informatik>Core dumpComputer fileElectronic mailing listBackupStructural loadComputerBus (computing)Computer wormEndliche ModelltheorieDiagramProgram flowchartComputer animation
21:43
State of matterWeb applicationRoutingSoftware frameworkCartesian coordinate systemWeb 2.0Physical systemComponent-based software engineeringObject-relational mappingQuicksortServer (computing)Energy levelGroup actionData managementOperator (mathematics)Game controllerDatabasePatch (Unix)Interactive televisionEndliche ModelltheorieStatement (computer science)Front and back endsOpticsSelectivity (electronic)INTEGRALSoftware developerRepresentational state transferTable (information)MappingSoftware architectureThermodynamischer ProzessDecision theoryDifferent (Kate Ryan album)Theory of relativityData modelStandard deviationWeb pageInsertion lossConfiguration spaceObject (grammar)Computing platformMereologyProgramming paradigmBitUniform resource locatorOrder (biology)Presentation of a groupRight angleGreatest elementExecution unitElement (mathematics)CASE <Informatik>Volume (thermodynamics)Numbering schemeAsynchronous Transfer ModeComputer fileMultiplication signMeasurementModule (mathematics)Program flowchart
27:04
Patch (Unix)Residual (numerical analysis)Software developerSoftwareInstance (computer science)Pattern languageQuicksortDatabaseNumbering schemeComputer fileBuildingMathematicsSpecial unitary groupMarginal distributionRepository (publishing)Web applicationAdditionSpacetimeState of matterTrailServer (computing)Table (information)Component-based software engineeringExtension (kinesiology)Endliche ModelltheorieSurface of revolutionObject (grammar)MereologySoftware frameworkData managementBitSoftware testingMechanism designThermodynamischer ProzessSet theoryOnline helpCASE <Informatik>Procedural programmingINTEGRALPort scannerWeb 2.0Module (mathematics)Object-relational mappingCodeRegular graphOperator (mathematics)Multiplication signCore dumpSource codeSlide ruleProgram flowchart
32:33
FeedbackMereologyWind tunnelLoop (music)Numerical analysisFlow separationQuicksortTable (information)DatabaseComputer fileConfiguration spaceSoftware developerOperator (mathematics)Sinc functionTouchscreenRight anglePrototypePrincipal idealComputer iconWeb applicationEvent horizonMusical ensembleExecution unitProduct (business)Web browserProgram flowchart
34:14
RootWorkstation <Musikinstrument>Web applicationDatabaseReading (process)File formatStandard deviationCASE <Informatik>Repository (publishing)Multiplication signMeasurementKeyboard shortcutBitOpen sourceTape driveModule (mathematics)Goodness of fitPatch (Unix)BuildingNP-hardHazard (2005 film)SoftwareWordMetric systemProjective planeEndliche ModelltheorieThermodynamischer ProzessXMLJSON
36:09
Uniform resource locatorWritingServer (computing)ResultantSummierbarkeitData storage deviceConnected spaceGame controllerAddress spaceComputer fileContent (media)DebuggerBitPoint (geometry)Computer hardwareCartesian coordinate systemResolvent formalismMiniDiscData structureDirectory serviceMetadataWeb 2.0MereologyFrictionDifferent (Kate Ryan album)Right angleRaw image formatCombinational logicData centerSearch engine (computing)DatabaseStandard deviationLevel (video gaming)Slide ruleSoftwareRAIDFunction (mathematics)Replication (computing)ImplementationInformationThermodynamischer ProzessProjective planeIntegrated development environmentComponent-based software engineeringAlgorithmVolume (thermodynamics)Physical systemFile archiverWage labourDivision (mathematics)Machine visionCondition numberComputer programWeb serviceGroup actionSocial classState of matterService (economics)Type theoryWhiteboardOrder (biology)Point cloudIdentity managementStochastic kernel estimationExterior algebraQuicksortRepresentational state transferSet theoryHypothesisGradientMultiplication signValidity (statistics)AverageDisk read-and-write headJSONUMLProgram flowchartDiagram
Transcript: English(auto-generated)
00:05
This is Alex Lai here in the front row. Alex is the DBA and I'm the developer and assistant DBA on the OMI Science Support Team at Goddard Space Flight Center. And we're going to talk about monitoring ozone levels with Postgres SQL.
00:26
So I'm going to begin by giving just a little bit of background and Alex will talk about how data is brought into the system during a process we call ingestion. Alex is going to go over some of the challenges that we've faced
00:42
and then I'll describe some ongoing development in the software platform that affects how we manage the data. So about 450 miles above the Earth's surface is a formation of satellites that are in a polar orbit
01:03
that takes them across the equator 14 times each day. And they call that formation the A-train. And all of the members in the A-train are Earth-observing satellites so they've got their instruments pointed downward instead of outward.
01:24
The caboose in this A-train, you can probably just barely see here, is a satellite called Aura. Launched in July 2004, Aura has four different instruments for monitoring Earth's atmosphere. The Ozone Monitoring Instrument is an instrument for monitoring stratospheric ozone levels
01:45
and that's our instrument, the one that provides us data for our system. Because these guys are in a polar orbit and they go over the North Pole each time they go around the globe, the data is received at a far north ground station,
02:04
like this one in Svalbard, Norway. The data is processed a little bit at the ground station and converted into something called an Earth Science Data Type, or ESDT. This is not really the sort of data type that you would normally encounter in a database.
02:25
It's really more like a procedure for processing a set of measurements and they're defined by Earth scientists, which I'm not an Earth scientist, but they provide sort of the raw interface between the data coming in from the satellite
02:44
and our data management system. And right now there's more than 900 that have been defined that we have to store in the database. These ESDTs get processed and are generated
03:02
and the ultimate goal is that the atmospheric scientists will use the results of all this processing to create atmospheric profiles and climate change models. So, Alex is now going to tell you a little bit about the process
03:21
of how the data gets in and processes. Good morning. Can you hear me okay?
03:43
I want to show this diagram to have a high level of the system on a processing system. As you see in the box, you see the data provider, previous slide showing the ground system capture all the data coming down,
04:04
and will be either send the raw data to us, or they do some processing, certain level of data, and those are the provider of the data. And they will get input to our data processing system.
04:22
And also, we have a large group of scientists community relying on output data to fit into the application and doing research. Those scientists will go through as a user,
04:41
go through the web interface to making requests, and the data gets delivered to the data receiver. And our OMI project operator also can go through our web interface to monitoring our processing and also submitting jobs through the planner
05:03
and see what kind of data need to be. Sometimes we need to reprocess data or any new product coming or some send is changing the logic to how we manipulate the input data and come out with the final product for the web interface.
05:26
And this diagram is showing, as you know, that the data basically have two ways to store the data. The real data will not store in the database.
05:43
Like the image or all the instrument capture data will be stored in the archive system. The data is very high volume. And the other type of data, after processing,
06:03
they will store in the metadata server. That's how the ProScript captures tracking all the metadata in our system. That two-way will trigger the ingest process.
06:24
Through the auto-ingest, which relies on the arrival of the new data, we have a logic to checking, daemon checking any incoming file stored in our archive node.
06:43
And then that will trigger the ingest process. And based on certain criteria, we're set to doing the process of those data. And also the OMI operator also can submitting any job
07:00
and controlling the ingest we can override and seeing the planner and seeing the load. And also we'll be able to also submitting job that way. And those are, maybe I need to explain a little bit more about our ongoing situation.
07:22
We're using PostScript almost over 10 years to tracking all the metadata in our legacy system. Back in a few years ago, we realized that the data schema need to be redo.
07:42
So we are starting another new development server, which you see down in the lower corner, the MDX is a new system, ongoing development system. And basically that system already partially in operation,
08:01
some application already pointing to that. And the idea is we need to have the metadata update on both systems. But basically the priority is to make sure that legacy system is ongoing
08:24
without losing any data. So we're based on that. Once we update the legacy metadata, another trigger will go into the MDX metadata server. However, from time to time we do have a little bit problem
08:46
seeing that are there any reason that we didn't successfully have 100% all the data get into MDX metadata. So that's why you see a box on the SYNC and our database process.
09:03
We're trying to catching any missing, any problem file haven't successfully ingest in both system. Yes. So in the pre-process are you writing to the same table or different table?
09:29
Can you repeat again? Yes, are they writing to the same table?
09:42
No, there are two databases that is two different database. It's not the same database. The schema is a little different between the two. Yeah, I'm going to show you more in the following slide. This is one of the challenge we are facing, trying to updating the basically same kind of metadata,
10:03
but in different format of the schema. They are totally different. We have different partition. All this have different. So I'm going to show you later.
10:22
In the thing I want to talk about besides the synchronization, this is one of our backup asynchronous replication. Right now this strategy we are using. I want to talk about the history about before we running NIO,
10:43
we was running it. We still having no replication that time. We are pretty much based on the PG dump to do the backup. I've seen our data getting very big and take time to doing the dump
11:02
and also the performance will drop while doing that. At that time we only have one server. And also after the dump and the data ongoing update and between the first dump and the second dump,
11:22
that window, pretty much the database crashed and we're losing those data. And the replication NIO is really help us to solving this problem. At least we have standby database and a little bit better.
11:40
But we still continue to do the dump but the less frequent dump. And also we dump from the slave and it won't causing our performance in the master. I just want to mention some of the, sharing some experience on the thing we are focused on,
12:04
how to make sure the replication successfully. This is a kind of area that possibility of the wall file did not apply properly
12:21
and then can cause the replication fail. So between the master and slave, sometimes the network is unavailable for a period of time. We have to get notice about that. And also possibility assay may deleting some wall file didn't aware of.
12:46
And you see in the previous slide we have a PG dump going on. Those is a copy can take a longer period of time to finish while the replication going on seems we're dumping from the slave
13:01
and they have some coordination need to be done. I'm going to show you some parameter we adjust to avoid the fail, complete the PG dump while the replication going on. Actually this parameter showing is a max standby streamline delay.
13:27
We set it to one to make sure replication in the way stay until the PG dump finish. That avoiding the crashing PG dump.
13:47
We have sometime need to use based on the PS operating system to showing the status on the house when the replication going on,
14:00
we are kind of monitoring those status and if they have too long a period of wait and we notice something is going on, the performance may have a little bit problem. So we are trying to have some two automated tools based on those information to alert the assay and DBA.
14:27
And we are also using the log and capture some information there in case we need to troubleshooting the period of time the performance issue. And also in postscript,
14:43
building table, system table, you have those information and also we can use those query to query how the log apply and if they are not move on, certain period of time, we also get alert and things like that.
15:03
And that's very helpful query table, system table will help us to monitor. And those are some kind of thing we also are aware of that
15:20
make sure our wall file is in archive in case we need to rebuild or restore and also make sure we don't perching the wall file before actually the assay applied it.
15:40
And we pay extra attention to those displays and also the monitoring the wait state, how long the thing and also monitor any dialogue situation happen and that's just an email we've been using to alert
16:03
We haven't have any pager and we try to not work 24 hours so we don't implement that. And because of our satellite getting all the data, input data, we are in archive server, the worst scenario we can always go back
16:22
and repossessing those data can be reproduced. This is just some notes about the performance issue.
16:41
How much data we have and currently we have about 10 cluster of the server in different stage of the development. We have the development server and we have test server,
17:00
we have operation server. And on average, it can be 200 commit per second, sometimes it can heat up to 10,000. It depends on the workload. This is the average. And also, accounting is very difficult to measure.
17:22
They don't have really the standard. We measure how big your database is and this is how some indicator how big. I just want to show you the one of the biggest table. We've got 160 million rows and the growing rate is about 40-60% the last three years.
17:47
And just have some chart, this is a legacy system we have. 2012, we are already growing, I think maybe expecting more. Sometimes it's very difficult to predict the growth
18:02
and especially, we may have a new provider join, they suddenly want to be processing 10 year of data, they can suddenly handful of data we have. So it's just trying to scale our system,
18:23
see if we can handle that kind of load. This is a new system. That is a bigger challenge I already mentioned about synchronize between, you see, it's very cumbersome
18:42
and also because the ongoing update of data make it difficult to identify 160 million rows of data, which one is the new one and also which one is different.
19:02
When we are trying to do synchronize, the script trying to identify the hardest part to synchronize. Once you find out which one you need to synchronize is easy. We're still working on trying to find a better way,
19:20
maybe trying to see any audience can suggest a better way if you have different schema, how to do a synchronization. You see this is the legacy schema. I'm not going to go through any detail, but just show you the new one will be simplified,
19:41
but basically partition the data in different table and things like that. Whenever the synchronization script is pretty much have to mapping the two different schema and trying to find the right query
20:00
to update the missing data in the new system. Some of the parameters I just want to show you. This list is not showing the complete list of the configuration in the postscript config file. I just want to put in the table side by side easier to identify.
20:24
If it doesn't may be helpful for the beginner, never use a replication may be helpful. The certain parameter, master and slave are different. And probably you can read that, it's very straightforward.
20:41
And pgCon have a lot of pretty documents, and have a lot of very detailed information. And as I pointed out before this, we make an adjustment because of the pgDump thing. We put in minus that. The trade-off that the slave during the dump will not be in the real time,
21:04
which doesn't matter for us. Our case doesn't need to be 100% real time in our slave. We're using a slave pretty much for the backup purpose, and also some long query, a select query doesn't need to update.
21:23
We're hitting our slave to offloading the load from the master. That helps us to boot up our performance in the master. And this is some other parameter we're using.
21:45
That's it for me. And I'll hand it back to Marty.
22:01
I'm one of the developers that's been working on the rewrite of our processing platform. And some of the decisions that were made in that rewrite affect the way that we are going to manage the data from here on out. And so I wanted to go over some of those things. The big difference in the new platform
22:23
is that we're doing a lot of decentralized kind of components so that instead of having a more monolithic design, we broke it all out. And these are web components like component X, Y, and Z here. And where it was appropriate, we also splintered a little piece of the database off
22:42
and integrated that in with its component application. And then these guys communicate with each other using REST. And I'm not sure if everyone here knows, I wasn't quite sure what to expect in the audience. It's our first PGCon. So I don't know how many people are developers.
23:01
But REST is basically using standard HTTP request in order to interact with a web application. And there's three parts or three things that I think are interesting to mention on how this new architecture affects the data management. And one concerns the web framework,
23:22
and one concerns the ORM that's integrated in, and the other is the system of database patches that we've implemented for doing schema updates. So I'll go over each one of these three. The web framework is, of course, just something that helps to facilitate creating a web application.
23:44
In our case, we're using Perl's Modulicious framework. And Modulicious implements a model view controller paradigm where, of course, the view is what the user sees. In this case, it's a web page.
24:01
The model is a state of the system or the database most often. And the controller is kind of a mapping in between the interactions that the user sends in through the view to do something on the model. And here at the bottom, you can see, here's some example mappings of URLs.
24:23
Again, this is REST, so we're using URLs for the users to do the interaction. They're sending in some sort of HTTP request, and that eventually has an effect on the database. And so in this example, we've got a GET request,
24:41
which maps to a SELECT statement on a person table. And we have a POST request, which would map to an UPDATE or an INSERT statement on the same table, and a DELETE, which would just map onto a DELETE. And that's all pretty standard, I guess, for a web framework.
25:04
The thing that's a little interesting about our system is we've extended Modulicious a bit so that when the web server starts up, it automatically recognizes the database that it's coupled with. So it'll find out if it has an associated database,
25:24
check the database for all the tables, and it'll generate all these URLs or routes that are used by a web application to do the basic CRUD operation. So you don't have to manually define all these things.
25:41
The new architecture also has an integrated ORM, and of course that's an Optic Relational Mapper. I'm sure everybody here knows that. And similar to the web framework, we've done some things to help automate the operations that the ORM does. So as the server starts up,
26:02
the ORM kicks in and also reads the config file, and we'll try to discover the database as well. And the ORM will generate the object methods needed for those CRUD operations. So we've basically got two levels of database operation mappings going on here.
26:22
So the web server starts up, and the web framework is going to generate a lot of routes, a lot of URL-type routes that a user would use or the application that the user is using is going to use, and these then make use of the object methods
26:42
generated by the ORM, which can then go about manipulating the database on the back end. And so a lot of this happens just kind of magically. Whenever you create a new application with this framework and you define your database, all these methods and routes just get instantiated for you.
27:06
Now, I'll say one thing. That can be a little expensive. So if the database is any size at all, it's going through all these tables, generating all the CRUD operations that would be needed to manipulate that.
27:20
But luckily, since this is a web application, it does it all when the server starts up, so we only pay the price one time. And so far, it hasn't been too much of a problem. So since we're doing sort of distributed development with lots of little components that can be developed individually,
27:42
it was desirable to have a way that developers could create schema changes as they're working on the component, because remember, these components now, these web applications, have the database tightly integrated in there, their own PostgreSQL database in there.
28:01
So what we did was in addition to the code in the Git repository for the component, we added the schema definition file in this base.SQL file. And then as changes are made to the schema of the database,
28:20
developers add patch files very much in the same way that you might add a patch to software code. You add a DDL patch here. And so new patches come in, and they're committed to the repository. Eventually, you want to migrate these things out
28:40
to a running instance of component X, or more exactly, you want to migrate them out to the database that's running within component X. And the software is going to do a little bit of help into maintaining the,
29:00
enforcing integrity on these patches to make sure things don't get crazy on that. These slides got a little mangled, I'm sorry. So let's imagine that you have a new patch that comes in here. So they've added patches. And as a patch is added to the Git repository
29:21
and tested during the process of testing it and committing it to the repository, the schema definition file is rebuilt. It's regenerated to incorporate all the patches that have come later. And an entry is made in this text file called patchesapply.txt
29:42
as sort of a bookkeeping method to keep track of all the patches that have gone into base.SQL. Now, when we're ready to apply new patches over to a running instance, there's an analogous table. You can't hardly see it here, but it's patches applied.
30:02
It's just like the text file in the Git repository, but this is a regular Postgres table that's keeping track of all the patches that have been applied to this instance of the database. So one thing that the patch software will do for us
30:20
is take the schema of the running database, check and see which patches, it can compare the patchesapply.txt file and the patchesapply table to see which new patches would need to be added to this database. And it can do a dump of these things,
30:42
sort of a hypothetical dump that you would get if you took the existing schema and applied all these new patches. And if you compare that to base.SQL, you should get the same thing, assuming nobody's done any unauthorized changes to your database. So it's a way of checking and making sure
31:01
nobody's been monkeying around when they should go through the patch mechanism and do it the way we've decided to do this. Not everybody likes that, by the way.
31:34
So the procedure would be something like this. You check out your Git repository and all the patches are in there.
31:42
They've all been previously committed. But this patch software is built on Perl's module build. It's an extension of this module. And module build is a software for managing the installation of modules normally. And in this case, we've just extended that
32:02
to manage the installation of the patches. Does that answer your question? And in addition to using module build, we're using RoseDB as the object relational mapper.
32:22
And as I said, Mojilicious is a rather new Perl framework that has gotten fairly popular in the last year. Okay, and I've also been working with Alex on some sort of monitoring tool for our databases.
32:42
And as he said, we've got around 9 or 10 clusters. There's several principal clusters that we have that we're really concerned when something happens, though. But we've got a number of others. And what we'd like is kind of a simple tool that provides something like a weather forecast of how the clusters are doing.
33:03
And so I created a little web app using Mojilicious since I've been doing this development work already on that that would read a config file of a JSON formatted config file and then throw this stuff up on a screen
33:23
in a browser, actually. And it just gives an icon for each one of the clusters that are color-coded to give an idea of how the thing is doing. And it's just a prototype right now, but we've got it reading the config file
33:41
and throwing this stuff up on the screen. And it starts a JavaScript event loop that will also create the SSH tunnels out to these databases and then start doing various monitoring operations. And that's the part where I'd love to have some feedback
34:00
from people because I'm not sure exactly what's the best things to be checking. I know there's various Unix commands. You can check some stat tables. And if you see one of these guys go red or yellow, then you may want to click on it. In this case, I clicked on a green one, and then it would give you more information. Again, this is a locally installed web application.
34:24
These web applications you could create with Mojilicious are very light, so you could just check them out of the Git repository, and you can install it on your workstation in five minutes, and you don't have to bother the essays with root access or anything. And that's what we wanted, is something simple to monitor our databases.
34:41
So any suggestions, I'd love to hear them. And I guess that brings us to questions. I don't have it posted anywhere, but I have it on my laptop, and I'll be glad to share it. This is something I kind of wrote in my own time too. So some of the stuff we couldn't get out,
35:00
like the module build patch stuff is kind of company stuff, but the monitoring tool has been my own little project. They call it clustericious, because we have clusters of databases that process things.
35:26
And the guy who started module, or actually the guy who started clustericious tried to release some other software, and we just have a devilishly hard time of it, because we're actually contractors.
35:41
We're working for the government, and they just put a mountain of red tape into trying to open source anything. So we're in a little bit of a bind on what we can release. Anything else? Yeah. How do you store your measurement data,
36:01
your archives, and how are they connected to the metadata? Good question. Maybe we can go back, and Alex may want to jump in on this, but I think he mentioned this back in...
36:21
Are you talking about the data, the real data from the instrument, and come down with the image, or that volume of data? Exactly, he's raw data. Yeah, those will be stored in the archive. We've got hundreds of nodes
36:41
and for serving in different data centers and things like that, we have a long history of the data, and those real data never move to database. Database only holding the metadata, which is describing when the new raw data we store, the location,
37:02
and when we take a file with the algorithm processing to what level, and how we classify raw DSTP, we are grouping by so-called archive set, and those information is the metadata,
37:22
which is the database, the searchable, all applications point to that, and how we get access to deliver the data, and also our database tracking, all the ordering status, and all the processing, even planning, when we process all the information out.
37:40
We have another bit of software. This is a huge amount of data which is stored in the archive. How is this solved? This is the question. How is this solved? I can answer that. Do you set up the connections to your metadata? Well, we have another piece of software
38:01
that we wrote for this. It doesn't use a relational database, and what it does, as you said, there are millions of pilots that are constantly coming in from the satellite because it never rests, and we do an MD5 sum on those and store them in a distributed file archive server
38:23
that can span across multiple servers, but the location is determined by the MD5 sum on the contents of the file. So you can easily just generate the address on the fly like that. Do you use any standard software,
38:46
something self-written? It's self-written, but it's not. I mean, we're still using a lot of Unix-type stuff. I mean, just do the MD5 sum and store it in a directory, and the directory structure is based upon that MD5 digest,
39:04
so it's generated by that, what you get back from that. So it's not too complicated, really. The problem is because there's so many of them, as you said, and sometimes you have to rebalance the servers a bit so that one server doesn't have too many.
39:23
We did have this in a database. At one time, we were storing the locations in a database, but we always had them just as files.
39:47
Yeah. So the web components mostly work with the atmospheric scientists who process the data after it comes in, and how does it? I think this is talking about the front end, front end
40:01
and back end, and how the data is getting into the database is the internal, pro kind of environment. And you're talking about front end, and the important front end is how do people know what stuff you have?
40:20
And they have fixed a lot of applications, even other agency, other scientists, they have their own project. They can write application search engine based on, as you see, I have previous slide is data receiver.
40:41
They can actually not only go download a file to that kind of user. You can develop an app, talk to certain post-home data and things like that, and you can automatically get the data when the new data comes in.
41:02
Pretty much we allow them to search in order, staging it. They can write application and have the same goals, the output file. They can feed into the application. That's how another way to deliver the data to. So they have different kind of channels even for the data.
41:38
Exactly.
41:59
For storing files, that is the heart and soul.
42:01
And there's a kind of a division of labor here that's created a little friction recently. So our team wrote the software, the stuff that does the MD5 sum and stores the files. But these SLAs over at NASA are managing the hardware back there, and they can implement the software
42:22
to implement any kind of RAID array or whatever disk system they want. And recently there was a problem. Files weren't going in fast enough. There was some delays, and there was a little bit of finger pointing. Well, it didn't do this until you guys wrote this crazy software.
42:41
But everything that you looked at, you tend to think that it was something wrong with the disk. It got resolved. But I don't know much about the hardware implementation. And it could change without our control. How big is this?
43:03
Archives? How big is this? Terabytes? Oh, many terabytes. No, it's, well, let's see. 2004 to now for OMI.
43:20
So there's been about eight years worth of data. Do you write your own solution? Because you know that there are many storage results.
43:41
Do you know why did you write your own solution? You know, the guy I worked closely with wrote this solution. And I think he thought it would be kind of neat to write it himself in a way. But we already had something in place, and it wasn't quite right. And not everybody agrees that we needed to write
44:02
something ourself. But we did. And there was some clever stuff you put into it. But as I mentioned, this caused a little bit of friction. I'm sorry. An application?
44:21
Replication. Oh, yeah. In the previous slide, we have replication. We're using, we're seeing this master and slave stand-by replication. But this is for the metadata part.
44:44
We don't have it. We don't have it. It's just that all data is few. You're talking about the metadata and the data that you can do.
45:04
OK. Thank you all very much. Oh, I'm sorry. You're talking about the metadata. Yeah. So far, 1.25 is one database.
45:30
We have two databases, and we're talking about close to about 50 people. For one of them, for one of them, two databases combined are about 50 people combined.
45:45
Five of them. I'm not sure that there's a good way to measure how big the data is. Right.
46:01
OK. Thanks.