How to migrate from PostgreSQL to HDF5 and live happily ever after

and what about that you should never do at ITER project because it is not a major disaster this was an extension want to tell you out entertain the

idea of what they learned from that and also want to use some of advice and now the politically issues fi face that the because I started the in this project as a newbie and essentially had to rewrite everything and it is not really easy even for the data point of view OK because I will talk about the so this market was that this is a very general conditions that probably even about information about the in

number of the foundation and we do research about this weights so essentially guys naked that nice cholera the maps

mingled in that you see on this

1 this can university in the

world because of the project when and where this is the map of Europe this it was a project called the shadow project was done at the p-th Institute Chile and essentially was the 1st to project where the open friends in which that anymore and the

product that we was used to produce these this nice column them up and when it came out at the

Jet and in nearly 40 several thousand plants but this was already done so we already have a successful like suffered project which was able to reduce these which is the result of that in the computation because that

1 of the 50 thousand points and you compute essentially the other tools system the brewery do you have the internet for anything it would be it would be thank you that and so that the idea that I did not stop the project and the arrival and to the

American and I had to

change everything was so so now what will be the middle representation the mean so I'm actually a physicist about because I came from convenient and are actually the domain should be a generality did in the past and want to give you a video of but many nice features that article as it should carry it I found by own like maybe you and some my interests the for sense if you have the multiple-inheritance invited and what they have to call about she make to the solution of the and I have a series of papers semantic classes with math and this was and old Flavius when is that by comparing the because the time was that this is this is the python was made on this I the time to write it as a benefactive looking at the content and so my contribution provide became little bit less I most of you know sort of the political modules that probably you are using the saying if you don't know it because if it is a dependency of sci-fi I think is the source of dependency of all I find it is independent of several through the web frameworks a violence and at end even framework I can discover what kind of model remember there was a framework that revenue the current goes in the same way with doing so I look in the source code is hidden on in the same idea and then in the course of scored you I found my that there was comments saying they came from the 1st half of the class of molecules that are probably are using this code that you either as a thousand 100 thousand on all of because it using the thing I'm very active and so I said that I started working as a vital development so that we need reversal long which is an experience I don't and then what for 7 years and finance theory we were going to release the constellation of financial risk and I'm doing this calculation of we use that it's surprising enough of the terminology of the legacy you we have the acid before the assets where the option of absorption so that now the essence of buildings and have to compute the this and damage is something disasters often send accuracy the most most dimension of the Convention on International Women's would get rid of not so the fact that the and the higher and chairman has written there of and after a while I became the demand in charge of the Great engine which is the simulation engine the competition and you even the models this is this is is is the mothers the scientists you are uh we produce these maps yeah all through I thank in this presentation because the of remember this morning in the of article made the beautiful women talk about that's variation the lost the scientists 1 side and the web programmers and the general programs to the other side and and I'm not exactly the emittances and being induced because I came from the I spent any absolutely no uh whether the opening up of the development is on the staff so uh uh I understand very well what he was talking about a student is in its operation and the separation between the words cause problems with because essentially in in our case where scientific

application back to this site model right this amount of called that officially because these to land in the class the components the storage of again the progress of the highest some parameters and they left the architectural this season of the of the potential for the program and they decide to do everything with the database which was the source of all the problems we have performance was everything was important distinction and what they did in these 3 or 4 years was to drop out of the situation and as we

and yes so I I I knowing whether scientific water because and from there but there was still when the business enterprise will and I'm generally I'm an old school boys so I like they're not much yesterday about the relational model like relations is and to say area of working with the world of the great analyzer imports because we need to start from that of the users and I'd I like this I'm very and so and this was silence the using this end and then I don't throw away your call the just because is so of this was my punch but sometimes you have to do that so I'm going to talk about you know what happened and why

it was forced to remove the to do essentially talked right of the holder thank you some teachers and that the because I always look the legacy coordinative because

what you do you're right small policy on the reform after you clean up to do most things you've got away with it because that would be foolish but was forced to do that in ways that you can rely on and so this is so it was my 1st reaction when exactly 40 years ago I got a job interview these foundations and there they around was discussing the guys that it would mean a lot of you know we use a positivist distorted a lot of protein analysis should be there I was very surprised

research will never know anybody doing to see you so many questions relations with starting with important well I thought maybe they had a major task and maybe they do the computation and memory and this is just the authorities not to really 1 maybe the other would be something that is indeed on the contrary there was no reason at all that the use of except that they were programmed with programmers so there is a new job so

everything was done as well but the diction and there was really the idea of the staff right this is an example of that tuple

retina by erasing the ecological that then the rest of the people the idea this and since this was a kind of a distributed the compilation you obviously various coming from the water also so you have a central role the master model whether of this year you say data of the actually at the beginning we're not sending the data to the waters where the of the computational send it back the beginning then all the surveys phrase that I have same time and that there where often at the base of if you could from the

database and that combining these CDMA which was coming from the water was somewhat obligation to slavery and story and editing was lots of folks cost because you have final and the workers if I remember microstructure of the parents at the same time so that we can it and then you you have and the waters of work right in the same

table is what you get and they're the worst thing was that was not the way to the capital of the database logic is difficult because energy was invented some of things then

what you do in this situation there I think so I I you know the only way we need to really well but what you the good news actually who was good sense lots of there's a lot of experience from moral as well explain the fact about the textual work totally bad in the sense that you it was impossible to the performance for it also was the new that if you

needed to rewrite something it takes a lot of time they're writing from scratch because if you start from scratch easy you know somebody gives you the

specification you know what you want to you you don't have to be decided that knowledge is you want to use it and there are many constraints is that the right but when you have an existing codebase you don't you must do the reverse engineering must the the what was that title understand the white and then you ask the following question so and there's a just my widely spread and they can tell you on I don't know this was written by programmer X which is not about you know not anymore he was changing jobs all of this was acquired with the scientist but analysis like these these things but have the slave so it is a lot of complicated story right and can right so you want to why saving his choice because after a while and and did some experiments that measure some and it is called the decay and the 1st monster was that the problem of selecting with the 1 you can see

that essentially where in 1 the we're producing ground motion field the emotional feel that is the shaking of the and

then there is a net quickly you know it's a obviously the began and acceleration of the things that you can compute is commercial-free of you know insulin along the points some there will be a big problem associated standard time to compute the emotional feel that could be but

the think time to save them with wilfried would be a lot you can actually was it'll be the 1st here the computation and then put thousand samples and this is 40 thousand seconds this is an example a around my workstations actually this situation the class that was much worse than so you have a situation where computation takes 10 times less than

saving the and then and this was not the divorce because I had then you have to read the data from the database and the problem was leading the the government of this because we have manually the 1st time we tried this computation the class that we have we are not going to say that it was a 128 behind the times for machines so that African American memory unit was not enough and then later estimated that it would have taken more than 2 2 terabytes amendments around itself if it was not until and so there were 2 options or you don't say that most of the already stated them in a more efficient way there and you can see here what seems you you will be

meeting that is an error working with you see there's only 1 column of the 2nd column here is missing at the it is not an error so this is the and gene OK we have a number from the results of the engine is is 1 of 5 and this was using the Committee's shows 2 and 40 thousand with the 2nd quality something that Diamond and what that people perform because the 2nd would be reduced to the 0 he takes less than half a 2nd so this actually is that about you can see because it is so there are 5 0 of the Riemann sums for the fact that many of you that have a sports or if you lose the HE justified so money and

actually must say that this this is essentially what the private when arriving presented 1 of the important for and 1 of the and this equation was much more because this idea that was for the supreme power over the this when selecting the viking said instead of being of many linear diversities cycle of the disease so I don't understand it and we have the be so this is already very much of the mass but if you remove the the basic ingredient 5 of them might and final model of the mind the means that when the

computation becomes 1 2nd when the computation we can find you can easily you you can do it computationally under which was impossible and I mentioned there's this kind of that's the scenario the emotional use in other parts of the course and when measure the speed of thought not states it is impossible but then measured

again and there was no mistake so you can deduce countries was a very rare thousand equations because of the jungle that was using the the observation that the soul is instead with the way will also them up and then the standard all measurements are actually there could be even bigger cases but I couldn't measure because you know you get tired after reading what they see that that you stole but the larger the computation of the larger the cluster there was the situation was also if Europe that the Bayes was mostly empty OK that is some performance if the database is nearly fully so that 1 can the space and then perform much loss of course because of the substance under the communities of the two step and we also have memory problems statistics so that I didn't expect these kind of incredible the results of the beginning because I knew about it to get 5 because the scientists of me actually and some experiments and there so that should if I was at least 100 times faster but I would not have expected 100 thousand times the reason is that when you do you with more experiments more the is writing about in the last case it is much more so this much so I wanted to do it with the stuff about you you can sometimes recorded there I went to my susceptible to various problems and told me look at the I'm totally convinced that you are right about the review contextual justification is just the the the the is spent and

I cannot tell the cytosolic value we have proven right again because we have to use in 6 months and actually there's another interesting thing that younger political legal minds gamers months before me the then analysis of the called it all the same thing for the for the Boston sending made but the same name by mistake went to in the and that all the developers that brought to the 2nd picture and of course they were not happy at all about the of the restriction and also this week the dating system between 2 a to because we have half the team in so we can offer lifting body and was about

you and essentially the demons to Wikipedia as a class years apart the mean so probably 11 and that's great released by means of a canonical damage and we and we went out of order in the recent past but it is very difficult to do it and efficient calculation that is if they have the comes through you in a format which is not that 1 OK because if you have to wear the table tables so if they are not actually went for the maybe structure with for the estimate for the use and the teams don't talk each with each other there are very interested in the battle on the world to willing to but you can imagine so what they did and did not think for 8 months not accept start you could have been easily and what was there Monday in a fixed advisor let 1st additional which I think is a is a positive thing was eating because the modernist stated that the more you want to change things in a certain moment it if limited time is at the moment you want to get at the end things can get done so with frustration is also by the we have various thorough we improve the section which

everywhere and not the problem but as you freeze the cell and the 1st thing I did that was to put in place and monitoring system John Wiley system was around a good measure of how much time it was speaking to you

and clearly useful to use it and it was not so easy to like it when the after that if you want to ask me any interviews and storage and relatively its cement passing because the states commodities and it's an informant was done on the computer science community finding and using the method used everywhere because it was done in strange ways and there was a lot of time in the cells 8 months but not the mineral but fortunately the

the the reading ill-treated and I don't I don't mean and the standard model body was 5 nobody was fine laughter people are left there it was made at the at the the company what wanted to move most of the relevant to by the end of the week and somebody left

2 fundamental job somebody will also the the contractor DHH other time meaning that at the end of the Middle East said OK you want to think I'm going to what we're asking Cambodia and the guy said the I have a friend that here so you know the the team referring and so we took charge of a lot of other things and was tested more than the president and of course was a lot of stuff which is reversed really not used for the and we decide what to do and we had been essentially me and my colleague regional underworld the let that the before and we decided that lets people all this because we can throw everything away and we change to them the in the

database so implemented immigration mechanism which is missing so as to the moral changes to the actual change Aquarius laughter toward 15 months more than 1 year spent fighting

with positive and we had something like 1 of the money the improvements over lot but do not find the money to the by the way we in the way that the release of the new ones you don't want to as a schedule that really is what the users and the of I'm not saying that everything was working around the world because some calculators like the the the 1 that I showed the map of Europe that plan didn't have the performance issues so that the water it where the atoms in the province language if we could maintain and this software is called the water finally at the end of this was in september song nucleation and he's more measurements and their allies that it was I could have included the intermediate model of what the process side was was not what they thought that and say OK can improvement the in fact that would to the fine but they need a factor of 10 times and analysis to to make this competition possible so uh

and the kind of there and you university if I was much better so because no we don't use and transactions of the detection was wrong because we didn't need that of basic we didn't need to do any queries with some displeasure queries which Erickson would be the need to deal with it so it was totally useless advice I wouldn't understand the threat from without and with this of window sporting because will

fall so we wanted to make this have open source software uh the so called all think weekends all so these we want using entities in the tub you go the reviews evidence problem and we want the soft landing platform of any scientists in the world seismologist going along the east and I'm thinking on this laughter not only in the class so if you want to understand that we must class the area that and you don't want to have assign these stories told all this and the fighting the configuration file since like so we didn't want to fall to prevent voice and the left side to what I a like the rest of the day so what people do do do most of use of money right his motivation with doesn't depend on the these I say excuse because for me it was clear from the beginning that at some moment the Boeing Model like rational would have been replaced on completely on most that might benefit from the so when I didn't uh it was a lot of 1st because the idea needed to duplicate essentially all calculators local variable and you can click to make sure that the missing and this and the 2nd best part of of the function and that and that should prior and made a lot of changes so here are some of them so now I want I don't believe the time for questions so we just leave you read slides and then if you have questions at this point business mean training time information and so on there was a lot of changes and the about management story is there is a positive and 1 of them is basically like this this is like that of is

used to store metadata information for the instance either compilation of usage of IT staff time stop time this creature uh then there is a table of results on the relation that is a table with the output the calculation and a table the performance of the English and that's about all this in the battle against race I this idea to define that if you don't know what

it should if find these these astounding because it really is unique comparison the software extremely easy to use and extreme form OK these are are some of the and and again

asking me after the thing despite and stuff and a lot of a lot of stuff to the idea that what entitled to read about in the windows of civilization to achieve 5 laughter automatic semantic because the beginning of XML Schema which was dealt another completely wrong India and not others can I am surprised because at the end it seems to be working most is surprising because it's such an enormous change that he went to present the content of paranormal anyway there it went when the sense that well we actually have a performance

which is really thousand times better than before the Reynolds destined for scientists and up they can I meet and then locked up in Boston Windows of was to much was to use words everywhere even on I tried what and so so levels I want there's something about what learned that may be interesting for you to do things

that I already knew laughter is the after this work has been even that in these things even more so this is a really essential monitoring the system in the 1st that and also whether problem would you invest in best because there was not a base so that the test you had to create a fake table is populated roles the 2 roles with this kind of stuff in the textbook and allow us to land and now the and 2nd because if you need to

find out if interior also all unit of a monster this unit test items the the original 1 because they were placed in the days on the implementation and make a change in implementation and was forced to to and the type replacement will function convex fine best that there are what you with the scientist wants because the science and then we with this model you should know that this is not immediately miosis finds numbers and they compare the numbers that what the scientists wants they don't want to process the implementation the that is in the training of this kind of stuff so also so you know I am all the mass analysis and someone like that I feel that I can use some technical devices 2 people and in that it there's rises 1st looting scene don't you don't have to match about performance things simple when they're seen 95 per cent of the time average cost without doing anything just them moving past the interests of all spent time in complicating your technology stack this back because I got a request from people turning why don't you try to use modern and that's why don't you try to use graphical processing unit when tried to compiling the compiler which are all looking at all of these but not my problem was that 1st I have to remove from the then I can think of these standards can complicated stuff 1st simplified and also always challenges this assumption 47 of this very complex geospatial queries so we were using postures and at the end the user about do we really need to these just pressure gradients and the answer was not given you furthermore partly because pressure planning and and there was little better than before so challenges such as

just because some code is there may be should not have been there so in my advice is I they must be people problem that you can solve here is that this is a the because good so it means if you cannot solve the problem with the to add the gravity of the supremacy cdataof because of course but the problem is not easy not keep in mind what are the probabilities and everything that you do is with the

goal of solving the problem in this period was moving was that so I could then remove was monster for political issues but I can think about and the bayesian some political advice yeah the the most there

you know when the area in the you can say I would like to change everything in the most currently I think I will tell you that way to the 2 he's right so is not a problem data is always and every time little more change that make sure that the scientist the users get a measure of the mountains or you can divide the same distance and now this is public and faster and the user will be out

this on the next time you propose would like this change there would be view because they knew from experience that what you did before was OK so in the use of takes time to these task because when you're right even if you are and like me of all of for 4th year when experience program when you're a new place you have to be so you need to take your time doesn't mean that you must say yes always the true that nearly as many as you can raise your voice sometimes without taking the issues that the human began discussing which of the main sometimes you don't reach an agreement because simply missing data technique and that freeze that's 1 thing I want a single repository inadequately treated using for me was the the issue of loss and how much we just had a single topology to it was speaking to the analysis of the method and the and this then by the end of the of a compromise self from foreign policy that is now we only to find out the single let's see sometimes you can find and and this forest there are several things that essentially if you did

some experiments retrieving of prominent of these floating point numbers so it's not important on the for the duty meet for my

bytes yeah using cycle PG book something like that and just divide because you know they're fighting for what is already 24 1 plus and the cycle PG more bytes of the jungle more myself for bias can become 50 by so can be an order of magnitude so this was surprising that the effect was from the but other experiences there that sometimes you you make a design because you don't want to run out of memory so and you know that it doesn't run from the command wouldn't let me state when we saw your computation runs 1 didn't get any seat of immediately and finally the program after name your oriented don't have enough merit renaissance in the sectors comes in the wake of the skull so it is best actually ability to free the events of the memory of the US and would this time the sum all of us and as a matter of after half an hour maybe you put to run parameter so they're not after an often I wanted to do or what I'm referring to and that was the reason or OK so sometimes it's better to have is

and actually whether you use woman with and indeed for its famous but just and then you you you can think action and the size of the change also it is more efficient the filterbank American memory yeah yeah that is understand a the story that the complete for the time of the year 7 minutes later that's sickening from inside the what they did for the code for the concurrency a decided they like concordant you choose sites in general because we had some maintain that components in this stage way so and where that leads to the ways of the components inside and all at the same let's use component for which a futurist and uh so I used I mean using the face of the to future send them right and making up the reduced and that they have a practical system so if I am on lot of useful useful if you choose to leave with the processing especially if I a class that

we use but everything was the similar the patented laughter was tested in train letters for the 2nd we will use using Jenkins and this would we assess really really data because that is problem that you saw the keynote this morning with this we use of efficient in many of which activities and the 6 months ago we we dollars you can distribute the called section platform our problem it's firefight BY as family the nice thing to was give you a segmentation fault the

sort of Attica mandated but any attention I have some aggressive through because of essentially for 2 years so you spend time trying to optimize this with was in marked only we knew this was but the wind I should have removed most especially the test because it you know I am on the site conservatively this is

something I think is that important so before the break if it's and if they break again fixed the figure it again again I think spilled it 3 times after 4 times that they see that this there's actually in my way and they don't need those and it's better to draw away replace them with something as after for time delay I should probably be fired and indeed that the 1st time I had a best friend thing was it wasn't in the and also spent sometimes time you what the feature was that after discussing the scientists and scholars and we're not not even independent features of the sentences are on the end is doing that I don't know about so and the rest features within tested sentence or inspected to be that important and not so this everything the line then

problems like very very proud of this slide because our code is because the said 1 after the other variable which is they said that although the libraries which is really by part about is related to this is the part of the scientists work on an opportunity of unfortunately and see here and there but the amount here you know the the the excellence of language this is the use of the engine and you the number of lines of code and this is something like 4 50 thousand to 70 thousand lines of code so in the the US because this is just from these ones this is because you to together with modern doubled the number of events recorded at this time because the status of modest most need more but if you look at the size of the engine which is my much right back to the thing I reduced the size of 55 thousand 2 and 45 and so on so was able to really after 3 years of working there this knowledge

school in the fall and that I'm very proud of you know the the intermediary intermediary theories that where the cations not science staff so that was more called but finally removed all the stuff and now there's no more in the field of view on is of 95 % and of course been this so let's see if I can not use anything correct nearly 2 years instead of 1 year they thought about the thank you