CentOS (community) Infra revealed

Video in TIB AV-Portal: CentOS (community) Infra revealed

Formal Metadata

CentOS (community) Infra revealed
Aka the Joy of running on donated machines
Alternative Title
Distributions - CentOS Community Infra
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
Arithmetic mean Process (computing) Software developer Order (biology) Archaeological field survey Reverse engineering
Axiom of choice Group action Code Multiplication sign View (database) Parameter (computer programming) Dimensional analysis Web 2.0 Source code Damping Endliche Modelltheorie Rhombus Covering space Predictability Enterprise architecture Block (periodic table) Software developer Moment (mathematics) Bit Arithmetic mean Message passing Process (computing) Repository (publishing) Right angle Resultant Ocean current Server (computing) Game controller Diagonal Virtual machine Electronic program guide Code Rule of inference Number Wave packet Product (business) Revision control Goodness of fit Well-formed formula Term (mathematics) Operating system Traffic reporting Proxy server Form (programming) Task (computing) Module (mathematics) Distribution (mathematics) Projective plane Library catalog Cartesian coordinate system Configuration management Human migration Graphical user interface Uniform resource locator Word Software Logic Mixed reality Network topology Point cloud Family Local ring Gradient descent
Axiom of choice Sensitivity analysis Web crawler State of matter View (database) Multiplication sign Client (computing) Function (mathematics) Order of magnitude Front and back ends Neuroinformatik Wiki Medical imaging Web service Mechanism design Response time (technology) Roundness (object) Synchronization Different (Kate Ryan album) Oval Damping Physical system Injektivität Moment (mathematics) Electronic mailing list Bit Complete metric space Tendon Connected space Data mining Arithmetic mean Data management Process (computing) Exterior algebra Ring (mathematics) Repository (publishing) Internet service provider Order (biology) Website Whiteboard Reading (process) Resultant Geometry Directed graph Row (database) Point (geometry) Metre Slide rule Game controller Server (computing) Service (economics) Link (knot theory) Open source Virtual machine Rule of inference Number Latent heat Term (mathematics) Business model Energy level Software testing Lie group Tunis Metropolitan area network Computer architecture Condition number Form (programming) Domain name Standard deviation Distribution (mathematics) Validity (statistics) Demo (music) Projective plane Database Line (geometry) System call Symbol table Word Uniform resource locator Personal digital assistant Query language Data center Video game Point cloud Game theory Routing Window
Point (geometry) Presentation of a group Group action Game controller State of matter Civil engineering Multiplication sign View (database) Virtual machine Set (mathematics) Public key certificate Theory Centralizer and normalizer Semiconductor memory Different (Kate Ryan album) Source code Software testing Physical system God Computer architecture Fisher's exact test Distribution (mathematics) Email Scaling (geometry) Structural load Moment (mathematics) Physical law Projective plane Data storage device Physicalism Bit Price index Flow separation Residual (numerical analysis) Particle system Message passing Data management Befehlsprozessor Process (computing) Software Repository (publishing) Mixed reality Point cloud Speech synthesis Right angle
Point (geometry) Backup Group action Multiplication sign Decision theory View (database) Data recovery Virtual machine Mereology Number Power (physics) Wave packet Semiconductor memory Different (Kate Ryan album) Email Distribution (mathematics) Matching (graph theory) Scaling (geometry) Arm Information Forcing (mathematics) Moment (mathematics) Electronic mailing list Shared memory Physicalism Special unitary group Maxima and minima 3 (number) Flow separation Category of being Word Arithmetic mean Elementary arithmetic Vector space Software Repository (publishing) Personal digital assistant Mixed reality Order (biology) Website Geometry
of it high now I was not expecting that much people on a Sunday afternoon With such a shitty weather outside the and problem how how how much of you are into in or or just sitting here because it's cool it's warm inside order be that's a good reason I mean that you have my vote and that there is a reason so we this subcommittee of this interest infrastructure and now the joy of running uncommitted on where would explain what lecture that that the topic so just a quick survey always using our and send us in the room on the job done so on who is more at so the mean and the reverse is in 1st cool and developers the rest of the cool so 1st thing the when I that's a
good question I'm am local diamond button guide you so that with a picture of a but in the way that was used to guess I that also explain why budget beer but that also that strange but axonal right like to speak English so well 1 I'm sitting by choice meaning that I really enjoy what I do our I'm not a developer of I had no other coding experience even if those days it turns to be impressed at your service and go to by code so we have to I'm a sentence of user for all time uh I would say it in view just due to the number of divided by installing result quite a bunch of machines and does and I'm also a project member so we covered a bit of history so you promised so that nice sure the nice then you saw show that we have for the project you should didn't get you should already going to cabling we have rules and we have a bunch of I mean I really need to take care are to be to get rid of all this should because of what it was the Dow's back home and along that of my wife would kill me so please feel free to to ask you to show that the move their offering will cover um and if Rossiter overview and also I think that with people from the distribution I wanted to talk to turn into a discussion with other people from distribution and just so we have some the guys there um I think I was to do was supposed to be a form of enthusiasm as well as their time and DBN low course so at the end of the talk let's this is about for example the problem Republic all have in common and I try to solve it with important from each other so the yeah once upon a time and sent also thought it's um into dominant for while for the demonstration that is already there really frosted resulted in dozen when we had the 1st with the central of you of course at a time if you remember and I don't I don't know on which people were using simple straight up 3 which was the 1st release ever the I want you guys me I was there as well so that was just a bunch of people doing that on the spare time for academic purposes on it was just for fun let's try to understand how to rebuild so Serbian many correct way and put all that together introduced into a distribution that is what a rel rebuilt so that last while some machine just being a basement or in a garage whatever just so be used to rebuild a bunch of social PM the our you know I think that would something like 5 donated mission so companies interesting and supporting us and despite mission would just doing or on the same time I remembered it was a nightmare because 1 of the machines sitting in the US was at the same time the web server the forms are the masters of a formula well all the things you don't want on the same server believe me especially when is thought to become successful so yeah so how did we manage that's been
years ago while 10 years ago configuration management was not the thing on the new kid on the block everybody was searching for solutions and now it was mainly of a bunch of local streets that were just so used to manage those machines will start to grow and I will cover that later the different inputs into between and now yes the term it was made is maximum 10 people working word for the good of time and I think that we just know which 11 wall interesting so um we were just doing that on all free time meaning that we the job and we didn't want to spend too much time on thing that could be automated so that's how we started to in this time in 2 dimensions and yet we started to use but it seems although the beginning of the physician ID was using myself inside inside of the of the scent of what it was but 0 the 2 of 23 I don't know who has used up of it to the train tree 0 the 23 year not that much and fruitful for just what with the history in 2007 in the center of the room we had lickin is just trying to explain people up but as was the way to go while just compare all what are the institution bit of so the current investiture is that obviously we are eating on all don't food so we have a bunch of a mix of CentOS 5 yesterday prediction that disappointed that's the pros and cons of an enterprise solution and we still have 5 of satisfied nodes in ruining critical um mission-critical applications for for you use 6 and we started to roll famine in current positive so um and the reason why we do what you to it's a slow migration because we need to remotely go all those missions was machine we ran exclusively on donated machine and we don't have always full control emissions so we have to install the machines and um basically to just get rid of the previous version we have mission a moment still being used and got out of submission of that was donated project 10 years ago there was a machine running Benton for with when you around that's what we had a moment not everywhere so we have a mix of course of old and new missions but we still have to take care of those omissions the and then before the cloud was the new kid on the block and was really the height and those were we had to consider for the centers infrastucture a thing as being world because we accident than admissions without any Israeli without any guarantee we had nothing is in that center I mean I think that we have 1 location where we have 2 missions in this in the center so not possible to do a cluster on with emissions sitting in my is any other 1 in Brazil which is what the situation that we have now so that was that was an interesting challenge to try to solve yeah so as I said we need to new there was a need to automate as much as possible including the installation of the operating system obviously but also configure the emissions were when they were running so yeah we might at multiple time from public to private and actually reminded of to but the 3 of the have something while we need to breed of love of at moment that means that we are running public message if all of and toasts on Sept 0 6 million descent 6 and 7 node with of tree that this and of that uh so but the agent 2 . 7 on simplified node you know why we call to migrate to puppets 3 all suppose 5 learning but full full for all you really yes that is acted upon so the diversion that you need to run but the 3 that something is higher than the 1 in the distribution meaning that that's also a challenge when you want to run an enterprise distribution sometimes the tool that you use don't care too much about so well that you have to gender with all those things but at a moment's it working for that matter the 3 that something can compile a catalog and then apply that on what the agent can operate at lower than problem so the family down we we've done in the past was well but as you have the 23 was quite limited so we decided to use something else of using and other than if it was a bunch of of things just in the sense of would use a version of the time was migrated to give no but now we want to to something more sexy on top of so we want to seperate basically all the modules above model from all the data on the viable that were specific to group of node or just nodes so you have multiple solution to do that on 1 of the solution is to use form and we try to use formant both of the but the diagonal for the reporting features and now also as a Nixon could survive so we would all the things into formant ordered revolve of their the competition uh dependencies everything is there so it's really easy and for and also a good API that you can use to interact or are we not continue that because of the formant pitch but is really a good product to use afford monitoring on with this dimension solution fast and 2 overseeing the simplest prostituting more using that it's all the job so we knew that is quite well and it would for a perfectly fine for this kind of simple in closer to that we have a moment so basically we have and all nodes and we have some of the logic so basically what you do the other thing that we were it was winter missing to have is that it's proxies because if for example you have a monitoring server that is sitting somewhere in the US and that you have to monitors the machines in Australia you can have a big big latencies so the more prodigy connected just um the unloaded massive server for monitoring task the better it is dinners nothing fancy but nothing special not at 1st at 1st sight by means using what we use because it's in the destroyed stable secure everything but some we add a special needs delegates something through other genus over 1 we use bawdiness for that which is not in the distribution it is but it's really easy to come by his argument evil um repository as well so why did we decide you sportiness the interesting problem that we have within sent us is that when we have millions of machines coming to America network at same time what can we do well we try to say that we want you to be redirected to the nearest mirror being 1 of the mirror in your country if you're 1 it's not a nearby country it's not possible at is a mirror in the continent if can only on the continent as the visiting minute you don't exist so we don't care so um for that we need is specific on blogging and now we decide to use GUI so by another time was not possible so what was the possibility of of some kind of access control is in redirections this things you can do that with you by view wide but you had to do everything from scratch so we just use for that
and now we just redirect it out we we went to binary of request so I will speak about the and sink node what we call again signals which are basically the mirror machines but also the 1st think target that we use so that external people can there seemed to compete uh center 3 from when they want to become an external the mirror same thing but I would just were into the nearest mirror and we don't want for example someone from America sitting on a standard of its link being connected to force of emission in Malaysia committed that the of its which is the case we have some mission for example connected sometimes 10 minutes at a moment's bawdiness is really simple I mean we like this principle is to be visible the board in this thing is just a custom backend at the moment it run on a very old machines but is able to handle something like 400 because of quiz per 2nd which is nothing I mean for genus but obviously not at the same time it's interesting because normally as a and user remember to put into that kind of query all cover that in when I will speak about the middle this thing which is quite different when you run young update you see that you have fastest Miro in your direct you and 2 other stuff that's the only real and not to want from middle that's under the 4 so yes obviously like every project every distribution you have other nodes like covering on Web services them the forms but tracker a million torrent tracker and pseudo so yeah that's that's really basically the so let's go for example 1 of the the raw that's probably 1 crucial I mean for you and use of its state and sink of integral the yeah we had moments of donated machine running hours and DOS and that their main role is only to fetch new releases signature-based dates and push them aspect of the last possible to John Adams so what we do is that from but system point of view we have to muster in to master nodes that are in separate at the center that we know for sure if can trust we don't want mission to disappear and pushing to and that's called doors rings 0 and ring once emission at this interconnected speed we push the mission and RDF there we just pushed to level tune were intuitions when it's done it's unable to at the moment we have more than 580 external mirror refreshing although the updates from us so that's from meaning that from a sad state when we have a new release where I greedy I when we do an aggregation on status it from those mirror from the military comes with control but the 1 that your pretty update from we're pushing a during several hours of more than 40 bits per 2nd so all that on running along than its missions the current status is that we have a lot of machine in he was some of them doesn't beginning was for strong and has a strong presence in the US meaning that a lot of people from and Austin industry and companies were in directed you have all of them we're order and said why we have thousands of service so we can give you want axis so please um ring solid do whatever you want with it we had a few in we could do with that emission ball missions and a problem is more in Africa on in Eurasia that's really where we have problems I'm guessing that it's the same thing for other the distribution as well because you know lack of resource we have only 1 mission for example for Africa which is in this in South Africa so that's that's a difficult due to solve for the moment and yeah sometimes so that in in 2015 we have Mission connected only at 10 meters per 2nd which is well if you consider for probably didn't a condition that you have a whole lot better and better connected and that and it's a server in the passenger so why do we we do we want to keep that I what European white for example center still interested in mission connected at us but would be a good reason for that yeah good point thank you so sometimes you have countries that are during problem and therefore really international connectivity that all is but thing so it takes more time for them to push update to emission manager connected that and again it's but in turning the country they're better connectivity meaning that yeah it's so it's slower to get updates to that mission 1st but 1 there is faster for consumer there we don't have modified and they just um they don't use mission of excellent used that mission because the Geo of the directed to the mission so that's better for for full do you as well so what we do is I told you about N Sync role so the militant a little world normally when you will install machine and you you run gen updates where you are not supposed to it's that Milton settles that or of it this good CN because we have more than 500 and 8 you assume other moments so we are supposed to hit 1 of the country you all in you for from from which you are on and using them under control so what we have is that we have in our status on machine that check in you know there's it's just to that in the it's verify those 5 in mirror in the times you justify every repository every readings in every architecture it even check that the eyes all the world but take some of the eyes of 5 so that when for example you want to download your sure that is validated so if you haven't flown from time to time on Centre little slide on it was that a every 5 minutes it's refreshed we validate mirror mission appear to others appear so it's done new the and for the people interested in your in uh know we i'd which which is generated a list of validated mirror for specific repository so if for example you would you use that symbol call request that's exactly what you miss going is just shaking Miller standard and the them give me a list of allow validated for my actual release and that's it in in your country and we use the same process we wanted unorganized image if you want to download lies image from the website it would just use the ISA redirected sent along what Service would just give you back a list of validated mirror in your country all the new my country yes question the the but no justice 40 ample I mean it for debugging we purposes in those proposing a in and I'm forcing that to me to be lobbied humbling not not done by young no um young leaders to maximize what Miller's process server will use max mind uh to guess from your IP supposing that is good because once and then we add user complaining that Amax mine things that I'm not in Ireland that invites him I'm in the UK so although sometimes well we have no direct we can think that much money there's something wrong with actual data data database but all you can as of reasonable but is on its so so
back to do the next missions mission challenge the I said that well all the mission that we are using alternate missions butts while in a perfect world that should be fine except that after 10 years I can tell you that we we have sometimes trouble in a sense that we we we have lost for example I we I don't even count the number of mission that we have lost over the years and for various reasons the sometimes um while there is another issue that can be solved but some problem but sometimes is something else that I will cover that in the next slide so for example we had to own we we've lost even just want to the 14 we read several and sink node so militant all of this and all the centers of all the nodes we had to migrate wiki 3 times during I mean in the last 2 months after my recusal just 3 times because yeah it's not possible to to shenanigans what I've almost shoes on those the machine what if it's a sensing note what we have plenty of when it's a man you recall that you have to migrate what he wrote the best and you hope that those mission those people with tendinitis otherwise you're screwed and that's 1 of the risk we have sometimes so we try to obviously and try with uh to my magnitude and company into the mission that we trust for a long time between node at for sure that there was the so if you put in just a decade of service director marriages into the 14 the meaning is the backtracking planted them so the little demo among the most that is known to the middle of the crawler that's just on the item migrants so well that's you have to use of motivational wise wall it's game over so back to the agenda of the admissions sometimes they don't even value that's all they don't want to pursue a more so you at a contact for example back into the line 5 from Mr. B at companies sees a while I I really like what you do I want to give you access to emissions sitting right at the center find but what what happened to the company over the years sometimes bankruptcy and just go 1 Zadok suddenly said while yeah machine doesn't seem to be responding more and you try to contact them due to guide all did but the point of contact you out and nobody's answering the e-mail bonds back all worth you see that the domain doesn't exist anymore and then start the trouble the other thing that we had also is that I said I said Mr. B from companies C I told you that what we can you could use mission that problem but companies have been acquired by combining the and sometime by company pay but combining it doesn't care about resource so certainly is that we don't want to sponsor anymore I mean well it's not the business model so we don't want you to use the bathroom so by by the In the worst case it even in the best as a new Mel in what is you just as good at for the monitoring system which is also bad somewhere in between we have some interesting situation when we discovered at for example the company was at existing anymore was acquired by another 1 but was the last emissions the rounds we have no contact nothing no details when we try to read is that no no new mission to the around in all that sentiment is still does so you get since it is then a chicken and egg problem what can you do is use it or not I mean room so any absent under just commit disappearing they not so went for the company said we want to help you we want to provide to act as a machine the the 1st thing we do is that you wouldn't think you obviously and the 1st which is what's kind of distribution you want that I mean do you you do you really expect me to answer windows on let's be honest so um GeV is there that's that's that's give you system she with the SSH or just send you are just sentence just any might the public yet again used to inject but what we do is every time we just we solution means mean you trust someone USS emissions believe me is I've seen strange thing in in my life like Boston companies is that what you have a dedicated server the 1st thing you do when you log in you see well I would probably do a complete audit and machines and you see that the injects the on key directly into route authorized the come on you know please it all using different got all the override this package I mean that's their choice no problem I don't want I will not mention all the H all did story with a bag for example when you say that the reason those machine this out they're just using a completely different girl the disable everything like a that is it all just because they're bashing Geoseq ins into this the grown I don't want to support that I don't have time to just spend will hours just to do not eat it just it's a federal cloud even it's bummer cloud biting just resubmission the faster right kick-start its use you there for a reason so adjusting result from scratch and then what about computers human-computer what I want emissions so but we just stop always more it because yeah we have companies supporting us for now 10 years that in there can we trust those people yes can you trust a random guys and I want you to give you mission in my data center of somewhere in the tradition probably but if you cannot do you conifer show so what we do is that it's about trust relationship you want to establish a trust on long term that is to be proven so what we do is at least which start with some danger and the role well that's a little its of as a I always was non-crucial while this and I say that the mission out militant all it's true but it we don't run anything sensitive emissions the I mean that's just simple sir and all that is also to be designed so even if they wanted to do something strange from applying a young client point what a difference anyway so an emission just disappear with just a matter of just removing from the genus repeated back and then nobody noticed enumerated emission just a brief so we start from that's level another good thing to do is so we do the desk the response by because you know sometimes you would be surprised at therefore service together single service to get mechanism but you ask asked them something and because you are open source of easy are not the biggest ever I mean I don't mind that I got our it's a rule but it took 3 or 4 weeks to get an answer on a machine that you will need to rely on you know way so even if you have nothing critical even if no problem just test at the support response time and and what that's all we can what quantify the the quality of young to to support and then from with the time we know that true can trust of his we can show them and then put more crucial role I I mentioned by the for example we had to migrate of domain and this 1 looks at all of the records that you recall emissions when we decided to move as where we need of course to not just well bland renew randomly select all let's take that 1 because I die and educational but it was from a provider we showed that they were there they were supporting us and that's the word that's the human location where we have 2 missions in the syllabus we can move DAP back and forth on the commission so yes we can use those we have also all
the resource has sometimes on the mission and we all know like we just recently and I was for the people in the speech and the residue that we have on a small deaf cloud we're going deaf cloud so we have forced physical nodes all running in the decision on and I was quite a bunch of of the myth of memory and CPU and we are using just local so that is because well that's storage is always a bit of an issue and we just reading the because of this was cluster and present will think as a single so solution for the physical nodes of physical load up at the same time EPA rise cloud and as the rich provided so but it's for testing we assume that the mind and we use OpenNebula as simple solution for for that
so west the fisher within this and thus infrastructure we study the tread on the center that is that and if you have read the uh if we if you have read but there is a way for centralized subjugation more and more when the team was just 5 of us but it was clear enough just to deploy this city where it was needed there was no need there was a did it was not a need for central ventilation but we are just an opening up the wall thing and I don't I sure there yesterday that Tomas from certain speed about spoke about the CBS that's in the little region which is mainly um committed system we have a bunch of machine just being there in the college system with God you relies on on so centralized of indication and we needed true to generate a bunch of its Fiona's certificates so we need a common tool to do that for us we can't reliable but for that anymore it not scale and another needy so subsidies sport of the people would be able to I want to contributors to the set of project that the discrete-time account and then I would be given rights of several things 15 is yes or no problem but we can things so 1 last but not least some faster updates on your the we have and as we had you included we had this system and we had a strange problem this week mn ghost anyone here you have said that I think so people not aware of what I'm talking about you just go away and we want it to we are not racism and so the interesting problem there is that from a send us point of view that it was built quickly In less than 15 minutes it was on all the merits and a little note but then file 5 hours after that people were still complaining that it isn't the there why the promise that epsilon era I'll supposed from Aceh but we don't control the mirror so we don't know when they want to get out right so we have a problem in the sense that we're thinking about I mean if you have in an idea that I'm and all open and for that we think about using something more clever like a CDN all a message by us and that's that's so American just which just provision people conducive to try and take action or whatever but we still don't know we we are started to spoke with believe with that because the male manager there are just a rewriting the you know monitor from but ideas yes West yeah you said yes so you had a question or and advice OK is it ice f I was thinking about the solutions that yeah you got so we're thinking about it so I would be interested in I I know that we have some people from opens she again so if how do you solve that problem at the moment it this time for all right yes the final if you in all of the you know that the the the the the only of the difference time not some people all of so I suggest an issues may already and yes I've just the middle but it's basically exactly what we are doing in a moment and also what knowledge so I it with more than 500 and 18 euros to justify cross process where we need to reflect free um distribution to architecture for distribution and base always a bit extra multiple revised institution the trova the tagora process takes more than 4 hours to do that and to validate so I was thinking about something like an emergency repository well could just drop and by for just point people to the repository for those of us I really like what it was like it was the data and yet what is the 1st and of the all the all the and then you will get them out of the data that you need to know that we have all the a so the walls the the law and the state of the art the a about the that that's 1 of the solution you have because you control the middle East we can say well just just return militants civil road and we fine because we are sure that we to date we controls mission we monitor this machine but 1 then you have millions of machines it in the same 60 machine at the same time well it's a region could just yet so you have know most this this a the what was it word original many of these all the theories the that of points get out of this so yeah so that's spot that's 1 of the solution so it's a mix of let's try to but it's some up-to-date narrow and in the meantime just redirects to respect repository or just the males in the little notes so yes I mean we wanted to do was alive that they just using for example or and putting all the updates to just Millicent a little but basically then we are at a bunch of particles or machine disappearing from the mail network because those companies that you exceeded decoded that who you are allocated this small so that was even more probable than a a solution let's so we don't control well
go then admission is with some it's a challenge it's a good because this season us its community so we still like an order we we we rely on discrete emission of diminishing but we need some something to more of the Sun from time to time so I mean our debate around and again I'm doing that new the decision problem in the whole the of the of the of the of the of the of the all of the the union of the of the of the of the of the of the of the the of the of the problem of all the of the of the the the of the of the of the of the of the of the of the of the of the of the of the of the of the of the that would be interesting to continue the discussion on I was thinking about using some kind of scissormen cross distribution lists on which we could share information because it's all about the learning from each other because we are trying to solve the same problem that we all at the moment so that would be probably interesting discussion to altogether finally so so this question because Question Time to question the fact yeah this is going back to the time of the the yen are the fact that the money like yeah 3 how do we get there faster than we have the renowned from elementary point of view I had moment we have what we store a maximum of information if possible informant but on the other hand we still have an inventory that we just still with sensitive major like the point of contact e-mail telephone number is possible so we do that on when didn't it's a repository this so that is the it's easy everybody can access from 17 these so on that's what we use the training and you subtract 1 mission for was donated Desipri because sometimes you wanted to track if a company was a called 1 other 1 we had that thought is that we won we discover that because acquisition we had 2 companies that were that will merge into 1 and we to missions and does that the center that historically from different companies sold from that point of it was a lot of it was not a good idea to put 2 of the same role in this mission because it was just struggle by the same network a continuous so we just try to do a mix and match with tribal that into separate but we don't have any specific tool to to that and more of the question yes the of the the the the was a user you to the number of words in the in the in the back of the book and you know yeah it was the part of the reason yeah I mean this emission can be really interesting and we use the transition evolved when possible but where set of forcible when you when you say minority race my within same all want to force commissioners and others and on the same rack which would allow and what then west with the benefit of using the emission I mean you just need to item on everything but I so whatever it is a very tall or the commission of assumptions that it's it's just exactly some movement migrating with except if you say that you are just using this to be to be or whatever to just put completely division over I prefer just just but during their insulation emissions some them faster don't 1 over there are few the yes we have taken the debt for less distribution we have of user backups our arms but we don't have the means to the crucial role III the moment is more than the middle of contend for sites so in known we sold big difference when the Trevor simple to run that specific node in a vector machine with other guassian and the idea to this because 1 of them on a guassian possible while it didn't scale so we decided to just above it of of the word it depends Sometimes we were submission that with plenty of memory and what we have another mirror and 1 in this and that and all this and country we decide to use that 1 for specific with small or or like whatsoever for example 1 of those examples and applicable to the the 2 missions in physical also yes we use but not that much just because you know you have 1 mission there 1 which and is not like an old OnPremise but but that doesn't do a thing that you can have like what compromise clout we called because it's geo dispersed everywhere on the world that's 1 of the challenge we have about the movement of so I would like to have everything into a 1 sender another 1 for that as the power of the disaster recovery properties but is not the case some of the question all I thing I would I'm all the time know yes I'm out of time so thanks thank
you thank you