High Performance Network Function Virtualization with ClickOS

I come from the NEC Labs Research research labs in Germany and this is a of presenting the licenses work then by me in my FOAF set in C as well as some of the people at university of peers so in 1 sentence quick again to uh unit of processing so what's the motivation behind it a

long time ago some folks designed by the Internet and other people quite

as we talking but we are floating schools and also uh with attitude taken containing switches and routers resemble but in reality uh uh are commonplace amazement things they're very useful for um higher number reasons in the extent of where before uh were futures security such as firewalls intrusion detection systems monitoring such as the the pair

boxes will balances dealing with the addressee's obsolete exhaustion issues uh performance such as mediator is transcoders there's limiters and some others offer more these nature-like there's a dozen social what's over there and the I'm having some issues with the DisplayPort so you cannot see the exact recovery simple but but these new Watson's are

really useful but there are a lot of these advantages at the same time 1st of all they are really really expensive 1 of them with the books that I showed you might be mitigated + around 200 k

and its head with difficult to every features often often you you you get walking to vendors to were pretty far from where the features and so on so forth then difficult manageable from require specialized personnel to you know matter the box and ends you normally you deploy these boxes to keep in the peak late so that cannot be scaled by q and more white want um and so it's difficult to share so this devices among other parties and system outside of the market it's it's our 2 new players from all nodes on so tell me if we had this in some way to visualize this middle boxes

for and shift still boxes into the software is so expensive boxes

we would have mean mostly solve most of these issues we have you but it is still not clear if we can do it in software and really coping the coping with the performance that adding Walters can achieve so we so and so that what I'm talking about here is that in that such a proposal the

glass is the mean yes there is uh a descended from machine that runs on top of that from state model wrote so let me 1st talk a little bit about click think that is

an is if is a framework uh to design alters to in persist packets and has this concept of elements is artist elements um uh there's an operations the packets or received packets for example the population the science Mexican please detailing Ford's but pocket to a different interface the this elements and contained in a configuration and basically you me solids uh ideas as a user space uh application was as or as a kernel module this balance and exposed at their companies if you have a counter on an interface for example value variables and the the such profiles is the forces and and so for example to the country of account element to it just read-only and we said to reset the countries is for 1 for example for OSB compiled office elements excluding the ones that deal with the file system

and if you want to further extends the place you can just write new elements in its by the so an example also the configuration

looks something like that um you have 4 minutes on to receive packets into messages sent by out the overnight should the element that you know allows but this from 1001 uh in this nation 10

1 UDP packets otherwise looks at all and so you can see that it's it's quite easy just to FIL the packets for example so let's take a typically we have examined uh power provides but

it was a printing system slightly modified wouldn't above then and the kernel and applications and until In the players in the case of less we use many last as our best of criticism and just a run plate then we call the whole thing the courts so it's that the follows a trend of will always seem abuse but the presentation or elements and

rise so that's its application-specific operating systems the work was having a good system to builds kind of these mean newest apps and that used to build because of and see what we get is a 5 megabytes image In total uh in size and we had to relate the control plane so we don't have a slice profile system between the and so you had to wait is always and wires and so on and before the reduced will times uh as well we suddenly something like that 1 2nd and any get down to 30 ms with time hence there most important of the country contributions are actually driving then he beats for all but it's for almost all packet sizes so In this against what matters of memories but the ability forced to process really really high petitfours the talk will be focused on this uh last bullet points the let me just give you a 1

slides of what consists of told shame and and we call black will compile of newly this is such slightly modest and we think and could normal uses for embedded systems and so and when told that version and the compiler well although I P is on antigens that and some other utilities notes compared to compile cliques and 2 newest real we also uses toolchain to build so for example I per and the fast but injured generated this and that not all packages we expect to X reports of the obligations of of its our tools that is based

on we have borrowed tools that we don't use XCP excel or a sappy ones and we we design our own POS muscles that at which is basically using the Z libraries and new suite to generate the bindings for or tools for now is supported by the end of

scripts and all panel we just about and and but the musicians were OK it was it was not exactly what a musician was like a learning process but most of the contributions of having sold the would times were preferring old and sort the was answer implementation sheeting for 2 and enough from a different that attached the heart scripts really slow things down like 2 and a

seconds and further putting and compressed image file as opposed to visit actually the progress of the mistakes longer than would the itself the so that's it for the better the control plane

Fuller this talk is about to how we actually improve news and our main objective is to derive this but its right so 14 88 million packets per 2nd phoneme-sized by hits references TCP actors in suspected um all 4 datasets it's

the 810 thousand but perspective the so we the performance analysis of this the whole but anomaly have and that we have on the that attached to a note of open this vigilance which ends all of these 42 PM which is managed by admin many have the shared memory this fold used meters and then you get an event channels to extend modifications on that front and our clique environments additionally we had 2 elements so for our natural interface which is that and so but so that's the configurations characterful by on the so when we plug all of this together there were several what in its 1st million speech just this panel numbers change all the time so that this actually be better estrangement but long time ago and we did this analysis in settling once it was something like this there for my Pegasos able 40 thousand more than 3 thousand 300 thousand part seconds when we place net like in the tree and that's 50 and read the whole thing together just roughly 20 25 doesn't practice was 2nd which corresponds to 2 to 3 weeks of the troops so there 1 of the issues for 1

despite and 1st of them all is the bridge that news um uh afterwards it relies on the Linux

was that which is not exactly the same means but not exactly we really happen that traits popping up each time want to send a packets you copy that uh you need the proper page to the uh the back and the domain in this poppy although it's than in batches is really really expensive so relying on estimates and we you use so it is the links that so the allocation and manipulation of this sort buffers are really a really expensive and also non-US of back in front of front and driver is not exactly as majors as women's but functional so it's also roughly 2 times as performance than when it's n I rights was a mere like 100 megabits of performance and you get the so the status news that it's not you know during the

whole the making of front-end sampling uh small optimizations on the make environment in the start this by replacing the speech of I um on top of it valine ATM of right we need slightly fications of tools and that but

to supports multipage means this is work basically a portal to net but that's it's where from but when we and so so because are relying on valid in doesn't use as the means we remove that by commitment to the manipulation and results were actually lies with green eyes in that 3 times better performance and 1 . 2 million packets per 2nd for minimum size so before telling the very last piece of

news is small but Bekkerman at my net not that development in the and links as a of build that is a really is a fast at framework that's nice when rates for managers uh subdued non-quantum and my it's

it's roughly corresponds to 67 nanoseconds for packets to Philip containing in between suspected it requires the changes in the device on device on but these are minimal changes to support the net model and but the Albany registers and of physical memory addresses and the packet descriptors Arab export are not exposed to user space in any way these must be validated by the kernel when Paul for packets and there's not accessed by users space it's similar to what the net channel on his presentation where you map uh subdomain into his space and spoon we the that stretches copy packets and basically Paul and forces of bypassing the Linux host and the important well it on the other side was born

like a small extension to net it's of sort of switch and that arise for between vehicle ports uh 18 around 18 yuan by this per 2nd lately winners who is like 20 to him but this was 2nd but the graph on

top shows up a comparison of all the trees which has lived in this region in this region valor but this is small the the the following volatilises is between visual ports where is that the opposition bridge and the print tunics so we did this as a number of extensions to uh 2 lattice that's makes to the switch to modularize the switch which means you could you could call we you could implement your own switching from Chad and and having européen modules that extends the the the the switch and it's basically the way users which is like a normal netmarket PI uh we not normal application using at market here the the so putting all this together

we new rights read what feeds off the net but that's something we implemented 1 of around when is is a really small network that's that tries to mimic what's of the kind of this

the application we denote wouldn't we we move the extra popping which means the buffers uh are granted to the to the museum and so on the really protocol is not the same as in Zen-like you have request respect I response use you copy factor the ring notified about cancer and you um event channels I used to proxy uh that the pole so when you said that by continued to all new space in a pull that the ring to send the pirates out in this case we emulate is refined channels but the biggest problem with this is that the brain uh limits and other from France but told to show that it's these optimizations are viable we also implemented Linux um as as as complacent and the we

eliminate the puppies so I thought I'd to would to mention how much memory do we share the net proper Apophis are located in these pages buffers are on to the

so which means for that the normal ring size of that my which is 1024 you share a total of 2 megabytes per ring as of yet not the Irish the the only provide different things size a as a form of depending on with your for our people as when requires me you requires small about so basically the the way

that map works you with the issue uh I sit but to open it not the vice into you know you open the network not devices to an isotope to resist the interface not which attaches the Linux from the links will start then our back and we'll grants the front and uh the range and the buffers including the brand

references of the range of the buffers inside the ring slots the front and on the other side grants the uh the grants regarding the range and checks that that reads the sports signal to Matt the rest of the book and so to overtake us the and the minimum memory requirements of these the following slides it's uh 6 megabytes focal and we derive the then it's for the when this thing such as well so but me just show you some performance evaluation of our system I told you that those

views within 30 ms indices when we would 400 houses the bullpen bendable so we must cells slightly 1 300 ms at what time the main uh motivation is that you are able to react as the traffic which has been uh reached the peak in can expand the number of atoms know the machines in order to cope it's higher rates but I said rates like 10 20 30 40 things for example

so the that detective and evolves from tainted 2 milliseconds of tools 20 and 21 seconds the entry Satiates uh plea configuration that takes a wrong between 5 tool 20 milliseconds regarding the but the

performance we got a huge use improvements and we basically achieve uh around 9 million packets per 2nd for a society and it's uh for Ps so we achieve 14 38 million packets per 2nd which is 95 per cent of widening suspected this although it seems a little bit on the latest net version which we are about to 11 12 and you can see that as we decrease the

ring size we the prison throughput put those suspected the course of events and sense being more visible and but as you can see all the other packet sizes with Philip the pipe this was tested 1 hole and server of forecast between that's 2 magnets that he

gets we dual porting the and that and you mix it's score is assigned 1 1 city once a for for for the the rest of them you the so fast

after getting uh a the then it's we ask ourselves how scale from 1 it's so we actually run some experiments

to see if we could we were able to drive 40 uh for the more fortunate that with the needs of the children and so we have to it so the the bars R. it's following just the ends doing the by considerations and the lines represent forwarding so the answer actually possessing buckets and for the for but through the outside world and so we achieve 40 so roughly 30 needs um 30 needs for 4 4 3 ports plants this is only the right you receive so we mind that it's uh it fits full-duplex it's the double double of the performance of so if we manage the x 0 we are it's sexually 16 bands we don't we don't the we don't exactly scale as well forming suspect it's over multiple next again this was when the same set up appeared experiment uh we essentially quest for

them 0 and uh 3 for the the uh each of the the course has 1 week then the all the Internet's essentially otherwise otherwise they wouldn't make sense to stop and there'd been down Table to processual packets so there was a it's fine grained uh turning on the internet

so doing extensive performance the also gains a new improvements but disclaimer this is not this is not exactly fair comparison so uh good box the highest of the bars are the highest was our using that method right which which is an

interview and supported so and you still they're all the water makes of the host but this CPI use the and

so this is actually my my favored graph is when we actually when me the waters of so the first one is just sum uh bike affording uh so it doesn't that's the bike the 2nd the 2nd group of parties that had an appealing is hits which changes source and destination modernist for Inspector packets at the 3rd 1 of is a standards compliant the corrupted the force is the final node and then rules at the time agreed not speech as think 10 flows with randomized words the us up to be as which for those the weights of the 1st half of the DSL suspect subscriber any does the PPP diminished because we termination it ideal handling and session management of a load balancer a full wanted that matters that this is about the flow in inches indexing system just the checking that the contents of the file when it's just 5 rules after

that's since this is indeed the walks the Saturn system that it's actually important to have a little late and overtake OS machines and just 40 uh microsecond of the way to the the whole traffic and we compare it with other systems and the almost called we don't use the way and we did the best of our efforts to put the KVM and a so the lowest possible fans we also huge improvement compared to the link stop-signal knew that to it so this is because

the you really kind of solution to write to the packet processing we like candidates I will show you that I just right after and the presentation and albeit resembles uh to

moral between 10 and 11 so feel free to to go there and I ask any questions I will show you the more than once and and in the future work we investors exploring the performance on the systems which is not exactly that would have doing high consolidation of these something like running 1 thousand 2 thousand 2 thousand units of uh we did some experiments in our there to to K the answer on a single machine and in doing so the change of this is emissions fighting this from functions that we each other and to toward the war a packet operations last and not least uh in the sense of it not leave it to the trivial resource that we were there how you'd take Bazerman presentation on ministerial 2014 what request and review open-sourcing around that time which is equal for and me in that comprises mean about system know as those that are of the request for comments on the back and front and lovers and more things to come so will be less likely to questions the without B this of the of the union of all these people and so on we thank you of work they this the continuation is really simple just the quizzes a TTL on the ice in the past just that is just to visualize how the configuration with just the quizzes kill enforcement of of how bank is is that we can turn off to what the fuck this is that yeah yeah yeah you have nothing to eat something that but

that you know what say is that the use of yet he but it will take you she was asking that what the configuration visited telling what being paintings on take the and the steering on

and yes the end of the this is for just 1 them with yeah In case the the study of properties of the is yeah you know that some of the problem all right and know the that actually right now it's improvisatore it is only a few yet exactly so not is normally the 1st makes a valid is a speech uh so that can afford to get induces a nuclear so in this case is really quite simple if you will to so this is from which a result um the and we had we made some pretty which of these already on the this event so you can attach any can that's the host but you can extend this switch figure on switching functions and all that is obviously the questions and the the the the the the and

so on and the that the rates the rates unchanged here uh so for example the mailboxes is that it's

actually that's for example that the so the middle the what's this 1 wonders and machine 1 of them into angles with normal disappear went by the so it yeah so I'm not sure if I still

have you don't 9 minutes and I would like to show

you a demo the so this

them all in this the so I the resolution is not exactly point good but the this is is to show how they're just to show the good times and we the the so basically of the created them over there and caused was is very similar to itself for example and basically creative we're here and we separately configuration Adam and let's see how it so this is basically what 1 and and

the we the the now on the other machine they're connected back-to-back we just been Vivianne's perceive that actually work you and the to Due to the saying not a single so everything and every single them as a single relation instances and being so that's it there more than was like 0 on demand and he's a single image to be transformative and think for the man and you can feel that is not the of the there at lunchtime and tend to 11 the the fit to In the

end and you In the name of it all GOOD changing the amount of

of the the the the the the the and I and that the moment that the and

the the the kind of thing and because of the and in the

last week and I got to the end of the year and

we will the movement of the new thing and the more you think of when you think about a lot about the state of things

going on in the past and what you environment and so on and what I was that just like

the way in which you would want to know some of the things I configured and and the variability In the present

in the in the in the last part the the the the

kind of people you know and OK no problem and remind you of

what I'm going to do that you just look at some level of trust in God talking to the right and the the the

the the the shop this last the I'm just going through the the the union of the 2

and and what kind

a the kind the the and

all that get to you and it is something that you you you don't have the time and

and realized that if

you were to be that the the the the the the the the

problem and in the

end

the work of the mind and of the mind and you know the

and you and

I think we can and the and the of we it was

sort of that in the past you know something about in

a the and this is the the title is this would

usually look up connectivity 10 minutes before I and have a lot

of valuable clock you knowledge in the

the he was the 1st of the year in there and the

of the and

what you and we and you need and

and the and I

it the

it who and this the end of the the and the and that you we think that the wind and and the the state and in the the OK you wanna start hi my name is Yolanda work for our net on community the Florida and I'm working for the scheduling being then the antidote about you stimuli with a number of that involvement basically we will talk

about the problem of scheduling a little bit then will go ahead and talk about novel of future scheduled concepts and then will be diving into orbit it's scheduling and a lot of samples I think the best way to understand that our through the samples OK you write your hand you heard about the work and I guess a lot of you are too high our so already for those of you are no doubt those of you don't know what is over or it is there and management platform for the ends based on Indian my provides can handle thousands of the hands have blood snapshot like storage migration last year migration lies everything is like physically and sorry we support also advanced active configuration for those those and a lot of storage connections Sun NFS glass the basically let's see what we've got a long time ago the got the questions from the user's list I urge you to use that list if you have any questions about over the quiet no quite active how can I define a maximum number of running the endospore holds if it's pretty

trivial but we've been supported that the then so we get back to it

later just wanted to say that you should use the use of this OK what we had in order is that a long time ago basically we had to distribution of the readings for uh running in my reading the and when we selected

the and when your energy and then we selected a whole on and CPU load policies the the even distribution that has to be a low-pass they're being and then it's pretty in all a limited we have only 2 Distribution Algorithms and we can construct the user-defined 1 basically that is the then then the in which they have we take a low with looked at a number are scheduled which is we which brought thus filters and weights very simple and easy way to filter there to schedule the EM on holds basically the federal very cops and clear logic that gets a set gets a set of on the left hand side and then we ran the feaLDA on on them and then we applied weight weights on top of them and cool slide from normal scheduler the we collected we collect a set of weights then we sum them up and then we ordered the holes that we've got let's see as soon as simple sample for almost scheduler In this is around filter written in by python basically it's a very simple method that gets a single whole stay out and then and a set of properties and then run really

simple called that either

the decided that this this house is and the scheduling fossils are extracted excluding excluding the from the scanning process OK according to to they drum for that the and and OK when we look then at the end took it to the data center this virtualization which is over we saw that we have a little bit of a problem with that concept because in in an older overlap each filter and weight is applied on a single holes and we have a larger concept of their class 3 you vertical called migration domain so in that in information domain each so each of the on holes

can be migrated to a separate and also not migration demand that cluster and we have another concept of load balancing a for that cluster and also a policy compare which the user can decide and creates on policy and applied on the last so let's take a look what we had in word basically we can create internal and external filtering weights the internal parts is 0 and for better performance we people within the server so we get a quick access to the DB and originally called of all the filtering weights from the previous forward back then with when we didn't have shelters in way well migrated into that internal scheduler and the extent of debt you basically all the users can support and extend any we apply all the filters that they would like the future and and weights on all the holes in the cluster for better performance and we want to have a better grasp of how the uh the class their behaves we have a

container is called laughter policies for you for each cluster we can define the set of filters set of weights and a single load balancing for each class the and and with support custom properties that you can and it's gonna of passed through through to FIL internally the this is and that some of the new model under the on the left hand side you can see a set of posts within the cluster then we applied each filter chain each filter 1 on top of the other and then we construct the way table which gets us the result of the selected holes that you have to Scandinavia mom the a we have the same concept that

you it is no filters the existing logic that we have

seriously which world basically validation was and migrated

into internal filters and we can extend it using extensively you in Python using the external scheduler that I will get into later 1 tissue show you're really the easy sample of

how we can use filters basically this is a bit children Python the name of the class will be the name that reading that

day and then the server gets the the properties validation of basically a set of property that you can add within the server and then the filter we get it and this is the name of the filter and this is actually the signature that and you need to override in order to extend of In to extend a filter to add a filter to the system this is how you get this is how you get there custom properties from the heat within the filter basically ident only when the filter is all about here you can see we get the time and if the time is within the interval of that we get from the of the user then we friend they're with Brenda holes returned all the hold that we got if not just every move exclude all the holes from the filter that's kind of bad example the let's talk about the weights it it waits at all the fields that all the holes that the fast-food filters we had a pretty find weights that date 1st 2 are a the CPU load filters than than in the adult for we

added a lot more weight models and so the way models which is kind of

easy now to add because we have a new architecture each filter can have been have factors begin for Eric either through the sort filters each weight can have factors and their character the fact that the weighting factors we can also at and external weights let's see and now sample

In this sample we use a connection to are 2 at the server using in Python masticated that we have this is the k is backward compatible and stable we connected as decay and then the logic of of the weight is basically we iterate over all the the all the holes and we append to elicit toppled the whole state the and the the weight of the of that holes here it's a little cut but the holes the weight of the holes is the number of active in the answers on that tells so if we have a whole slew basically what we ordered by and on the number of running the and the thing that tells let's continuing have about it the load balancing that that's the 3rd model we have the in the cluster policy each graph suppose you can have only 1 and load balancing logic basically you can do whatever you want within the load-balancing connected ES decay and and chat on anything basically but what was appointed dominate is that load-balancing logic will return at the end if the holes I was shown in the sample later on In the end we will Meyer that's the end a single b and many women Sara basically more are safe to to migrate the single the and within a on is a period of time not called immigration last in the in the use all of our resources for that really not safe to do that we also have a pretty fine load balancing a within over the 2 a legacy ones on CPU and now and in 3 . 4 or even distribution 1 that we even the and distribution that we didn't have the same then balancing sample on the show as shown earlier the is we get the same number that we want to shut down all the the ends but in this example we will actually do that and not exclude that In prevent users from mining holes aviones like in the future examples here is show how in that all in the wake up call we will basically in get all health and if its wake up our will activate all the host if we need displayed and there is no then we will connect using our as the decay and get hold of the ends from that tells you will stop all the years in the deactivated holds say and then the example the the this is how we use it internally to migrate to the ends the you get a according to some logic you get the overloaded holes it is called me but then you select here it's random in the prose the animal that tells then we actually printed because viewing the we're using city I'll do to get the data are a from the from the model and the lieutenant and return a right listed holes which is kind of a filtered the 1st filtering we do before we activates uh the filtering weights and the normal process that we do basically as I said we have a class policy which is a container for all the filters and weights in a single of about a nonsingular logic and we have 2 optimizations for cluster policy the speed and overbooking basically we're under each time schedule at the end we run it 1 by 1 because you want to prevent overbooking the same round it we want to guarantee the same resources for free GNU if you will the and tried to schedule to the hands together we can fail because they the book they both the the same resources so basically this the optimisation these to exclude dead and weighed

of the variables so later

on that load balancing will be doing for us and bond found that the class of half and overbooking is to me it's there all island have just be able to power lies with the scheduling in Brussels let's see if we can have to show this picture is within a Word the it will carry on and on and the

it

induces a lowered the VM is you know because of Wi-Fi and the piano and the the M is unknown it's running someone and here I could

figure to yeah I can configure new class of policy you can give it a

name like the shut 1 if you I can add they external filter that I added this is the shutdown holes filter do they enabled filters the the here is the weight I I will give the optimal for power saving that's try to other data although the and for within a single almost as much as possible then I will Isolet deducted advanced analyze you created earlier I can give it wake up like a day in the and shut down our it rests OK should be created the criminal 2nd in the the

OK let's go back to the no no shouldn't take some time but we have do not connected to the you the I cannot the something happened to the began doesn't like me yeah I have strings have features that you know and the you may need to work so can you think of it as in strange iteratively maybe you were we here

in the the no theory the the given mind we

show that then you go to cluster we know how think and are I going

the good I creates draft policy that I showed you the the then I created to a cluster that already so all the holes within that laughter without going through the last policy OK

let's talk about how it's implemented there in no back the it's disabled by default share whoever wants to expand to add filters should be able to know how to install external scheduler the sounds scheduler in a separate process is written in Python we externalize it because you want to guarantee the engine safety nets you know if the user rights according to the dangerous thing that the system we want to allow other non languages well what if you know the end all written is written in Java sold and this model is written in Python and by phone wants rule uh support to staff which is gonna warm scheduling of the service it the the the the it's a separate copy and you need to amend story and how it works basically it's initialize it rounds it reads from now on local directory all the filters the weights and balancing logic that you you wrote then it's sprout publishing down IPI the engine the server read it and then it waits for the calls from the engine for filtering weighing

balancing this is how it looks like when it's loaded in the field and lower bound gate head back

to the users this now you can easily write that filter that in maximize the number of running the unions 1st holes the the early it is something up the we support the easy Python bindings for you to use it for you and scheduling uh you can can manage a separate policy for each cluster for each group of posts and have a very gendered comes out over gets new models get the questions this so that will used but I have to be to read what I have a lot of related to you ladies and since you have a lot of experience in a very very

good right now you may have to go this is the sort through

the history of the state or right which stores the creation of the region and the other possibilities that they were not suitable for example the dual of this would be a guide to find all the the of what we are doing so yeah you the basically because we have a memory of you may be out of school so that these all through the use of words that you have the idea it you can think of whatever you know it you are her question is I think you mean with any you know when you stand something you can do whatever you want when you send you know to old when you basically can connect every provider use any of the cases like so but this provides with the the engine is all that you us how memory and and CPU if you want to connect to other standard provided the objects and with the the people on the the right and so if a few guys in the also talks open but the law and they will asking us to connect the scheduling of C system that is more it's the are in upon tools for example they can monitor all of the dual and defense speed and they can actually predicted that if the sense to its friends speed is constant all zeal and this circuit and also is high the course is going to crash and burn in a few minutes so what they ask for masters to give them a list of all so they can actually be some of them because they have a little more information than walked over to have and there are also many other examples which of those similar of for example Cisco's very similar concepts the actually want to blacklist some of the whole sky fills the middle it's going to go down to the lotto Villasenor also metals in very big to close this is actually highly 1 the the the value of the policy committee details of all 1st of the things we have alright alright you know very well OK all right for those with whom I worked for the next thing is if I have this storage you information or information that had to do with the quality which storage perhaps I could utilize our moved during showing in a storage for our storage most of the victims of 1 what the other but could be nice to fleece and long with this and all the

the time it you it could be the the more questions the was 1 the culture of the the the thank you be fed

the and you you know you kind of

you know the end of the use of it we we