Salting things up in the DevOps' World: things just got real


Formal Metadata

Salting things up in the DevOps' World: things just got real
Title of Series
Part Number
Number of Parts
Santos, Juan Manuel
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Juan Manuel Santos - Salting things up in the DevOps' World: things just got real SaltStack is a thriving configuration management system written in Python that leverages YAML and Jinja2 which, by now, probably needs no introduction. This talk will explore Salt beyond the minimum required setup, targeting developers/sysadmins already using Salt, and those considering making the switch from other systems but wishing to dive deeper first. Attendees should be familiar with configuration management systems and practices, and comfortable using and reading YAML and Jinja syntax. ----- There is much more to Salt than the basics. This talk will go beyond the minimum required setup and will take a look at Salt under the hood, which will appeal not only to system administrators, but will also be more interesting to developers and to the DevOps community in general as the talk progresses. Topics include: * Introduction and basics review (master/minions, matching, grains, pillar) * Salt Mine * Syndic node * State modules vs. runner modules * The Reactor * The Event System * Salt Beacons * Salt API Attendees should be familiar with configuration management systems and practices, and also feel comfortable using and reading YAML and Jinja syntax. This talk is targeted to developers or sysadmins already using Salt, and to those who are considering switching to it from other systems but wish to dive deeper before making that decision. After the talk, attendees will have a better grasp of the more advanced possibilities that Salt brings, and be ready to apply them to their use cases.
Server (computing) Computer file State of matter Code System administrator Patch (Unix) Multiplication sign Electronic program guide Virtual machine Translation (relic) Auto mechanic Insertion loss Mass Food energy Event horizon Video game Term (mathematics) Hypermedia Core dump Representation (politics) Control theory Proxy server Address space Enterprise architecture Normal operator Thermodynamic system Product (category theory) Information Interactive television Set (mathematics) Configuration management Functional (mathematics) Local Group Estimator Metric tensor Subject indexing Data mining Arithmetic mean Computer animation Personal digital assistant Self-organization Right angle Whiteboard Pressure
INTEGRAL View (database) Direction (geometry) Mereology IP address Thermodynamic system Web 2.0 Computer configuration Single-precision floating-point format Control theory Information security Enterprise architecture Product (category theory) Thermodynamic system Software developer Basis (linear algebra) Interface (computing) Instance (computer science) Connected space Data mining Arithmetic mean Message passing Network topology Auditory masking Order (biology) Module (mathematics) Modul <Datentyp> Point (geometry) Implementation Server (computing) Computer file Virtual machine Letterpress printing Mass Protein Event horizon Number String (computer science) Energy level Subtraction Standard deviation Scaling (geometry) Information Key (cryptography) Model theory State of matter Set (mathematics) Line (geometry) Directory service Configuration management Local Group Event horizon Computer animation Personal digital assistant Vertex (graph theory) Statement (computer science) Codec
Point (geometry) Trail Thermodynamic system Process (computing) Computer file Multiplication sign System administrator View (database) Bit Directory service Ring (mathematics) Configuration management Event horizon Image resolution Arithmetic mean Mathematics Computer animation Tower Single-precision floating-point format Vertex (graph theory) Data type Resultant
Complex (psychology) Group action Existential quantification State of matter INTEGRAL Multiplication sign Range (statistics) Sheaf (mathematics) Weight Disk read-and-write head Mereology Formal language Component-based software engineering Hypermedia Encryption Statistics Control theory Information security Area Thermodynamic system Process (computing) Namespace Bit Price index Representational state transfer Flow separation Arithmetic mean Process (computing) Quicksort Resultant Point (geometry) Slide rule Implementation Computer file Event horizon Theory Revision control Natural number Computer hardware Software testing Subtraction Associative property Units of measurement Newton's law of universal gravitation Authentication Key (cryptography) Information Chemical equation State of matter Independence (probability theory) Configuration management Word Event horizon Computer animation Logic Personal digital assistant Dependent and independent variables
Pixel Greatest element Context awareness State of matter INTEGRAL Multiplication sign View (database) Range (statistics) Survival analysis Water vapor Function (mathematics) Mereology Perspective (visual) Web 2.0 Mathematics Component-based software engineering Atomic number Computer configuration Core dump Control theory Error message Information security Identity management Area Curve Thermodynamic system Product (category theory) Spacetime Shared memory Nominal number Flow separation Degree (graph theory) Message passing Arithmetic mean Data storage device Telecommunication Order (biology) Module (mathematics) Website Cycle (graph theory) Resultant Point (geometry) Server (computing) Service (economics) Open source Information and communications technology Virtual machine Auto mechanic Point cloud Event horizon Rule of inference Power (physics) Number Workstation Revision control Term (mathematics) Energy level Medizinische Informatik Software testing Subtraction Posterior probability Form (programming) Dot product Information Demo (music) Element (mathematics) Library catalog Set (mathematics) Limit (category theory) Configuration management Cartesian coordinate system Local Group Word Computer animation Software Integrated development environment Vertex (graph theory) Dependent and independent variables Window
welcome to solving the problems of world and here is 1 loss all salty guide in using few but of you coming for those of you who been here last year I gave us all this is going to be a little more index so that's the started 1 of the OK I skip the the boring stuff on the way my name is 1 some I work as a team leader of the board engineer at Red Hat um I am also 1 of the organizers of disarming which is the Argentinian system administrators community and under there a lot which is a local working slashed that conference events that we do every year in 1 of various seems to hi users of a couple of years now of many regrets or we've already ranks which are you want to but then again at this cleaner 1st so that's getting a lady I am only a home user sold I have to include the code I have submitted about the patch but not much more and yet I only have 3 cases to prepare this so was like pressure right and my thanks go to your Python for managing to squeeze the talking so why so the as you may or may not know us us all to start configuration management system I think if you don't know whether is the pocket show unstable what only better and why do I say better because it is written in Python and liberties ginger knowing knows some people in the room might not like the but you know you don't use Jason if you want it is relatively easy to understand and it was a relatively because it has some complex things but what he lacks in simplicity of really understanding makes it up in being extremely powerful and giving you the fusion of control over what you can do with it so all this will be seen in the next and 1 more detail that frequently gets lost in translation of spoken production agents that the case you don't have read access for you not allowed to run the agents on your machines the SSH much like Council so previously in Python as I said and last year given this was mainly interactions begin with basic mechanics in terms and concepts behind sold a secretary Council has a massive media architecture where the master is the 1 that gives of the orders and meaningless are awarded to meaning it does so by defining states and high states of the states are a representing the state of the system should be in the whole collection of states that should be applied to the system school other core concepts a set of matching which means targeting your unions and determine which states of life to which means and finally there's the concepts of grains and brings the information sent from the unions to the master and appear information sent master problem of suddenly and I have to say this still wife and 3 support soldiers to by into the is getting there there's so many issues hopefully we'll get there as usual is not because of soldiers because of the dependencies when they were only 2 more concepts that didn't make it the last representation are those of the mind and the mind essentially get started from the is sent to the master a regular intervals now even though this is done a irritable the snow you see useful for metrics because only the most recent data that you collect is maintained another thing that might confuse you is that all the data is made available for and so when you you might get answer whether the answer of the same data from all the means of the same time which can be quite confusing in fact you might be wondering if it is in this migrants is in what grains as opposed to you know get that from the unions and the last kind of think of the mind that isolating more often of the studied there only a very few purpose for purpose of the year of the them which you it's not something that you would should do I also if minions means that are from all the Slovenians the mind acts as a kind of so there's that too OK there are 2 ways of defining which mind functions you want to collect from the and the case of normal operation of salts it you would do so the pillar or on the minions configuration files but in the special case of not using the agent estimation before you have 3 ways since you don't have the means configuration file you can I use the roster the pillar and the master of your and so quick example of what the soul mind would be so that we don't have much to confuse the promise you get confused looks like this so let's say 1 of the 1st target of Armenians in our what servers group how we're going to be a player mine mine function to gather DAP addresses often 1st you every 5 minutes so uh this we can later it was for example and in energy proxy confusion to populate the surrealist now I know that you might be getting baffled borrowed all
would change if you're trying to think of it the important thing to understand here is that should we add a new host to the Web server group we find minutes we can have its IP address of the HA protein information this is all thanks to the mine which we can compute this interval of before we continue since were ready mentioned that Seoul has a master mean architecture there's inherent topology to it so this talking about the most common 1 would be one-to-many meaning 1 master minions but of course this is boring minor scale this also scattering the plates so what are the options of how much can we toy around with this we have to of say more masters we have about multimaster topology and I don't think there's any information security guys in the room but if you are you all of this questions about who he implement segregation meaning could we use a great part of the infrastructure split it so they don't communicate with each other but you know there's still functioning soldiers from of I coincidentally nowhere in the right hand this has that this those questions with another question so what if we tried willpower soul to soul this there's something called the sending node Anderson and it wasn't intermediate of 5 which acts as a past the name of it is to control a given set of lower-level menu which means that in the case of a cynic node we're going to be having to the scenic masses the show you can running instance so the way it works is something like this the Main master which no we're going to call the master of master you will see even tho it's already a funny name in some sense or 2 Romanians to this indignant the sending node relays those orders to local master that is running on the same machine and then that master again your gets the orders and relates them to the lower so now always in that no it's actually called a matter of minutes and so I will this of course works the other way around when some of the lower level minions replied to many orders they go to the lower level master and to this and then up to the main and so if we have the mask which now we saw massive masters and can have as many as you like connected to it now we have a single node for example sending messages lesser means which can also have in the number of meanings but there good thing about this is that we can't even nest levels of sending 1 of the other and have as many as like so the topology here it's kind of to you the only thing is we're gonna have to ensure connectivity is where the lines so how we actually do this this configuration is quite simple sending node where setting the standard master directive of the each this should point to or main not and and we also have to define a 90 year because in the takes the idea from here and then on the master node of course we have to tell you that we're now ordering all master we're now in control sending but in the case of the Armenians they should have the role of the minister they should have the idea the ascending node In integration just a few more steps we're understanding and of course out of them unless there we're going to have lots of the key because essentially there's a new that gets charity so now you might be getting the idea behind this talk is to make you think of the possibilities you could have difference in experiments development QA production and you can also have a difference in this to comply with some security standards that you might have a lower than you might wanna come up with and just to mention it we can even though multimaster with this would have seen and many masters main I will not covered here but just know that this is possible so that's it for my understanding where onto more hidden also or 1st up here is going to be given system so what do you think about a system that course it's track events that's not that's not the only thing but the important thing is that events can be acted upon and so this is this is also the basis of the rest of resistance that we don't see In this talk in essence this is mainly a steering Q pop interface being born things to understand here is that every event has attacked which allows for quick filtering unidentified and events and also vast amounts of for arbitrary down inside of it which just information about the so with just a simple commands from the master we can already start watching for events you know it's not watching what's going on we can also use the Celica and to a random events that we're not just making it up and you can see that this would be tag those would be the that in the event that is that I mainly yeah Jason string In by 20 will be a dictionary because fighting also stand events from the point of view and if we did 2 things right after stand in the event they should show up if we were watching attentively the event we can see that there's attack if there's a something not another interesting that being I think it to make that distinction last year there are 2 kinds of modules statement the first one is the execution models and the other 1 is right so the execution models that is the main
kind stated using salt I mean something that is going to be run on the menu whereas the runner modules are going to be wrong on the master it's this runner models can be synchronous a there yeah the runner directories configuration in a master file and the rest of us part whether we put inside that directory to bicycle or more use essentially this essentially and then continues is we just talked about events any print statements that we need to look inside our modules will be converted to events so by if we do this instead a writer might you will get some like this 0 OK sometimes not quite nice but there's a so even though you can write down a modules and you're certainly what do so but it is tempting but there's actually no need that I I mean there's already a full list off from the models available and sold In documentation so feel free to check this out there are wouldn't it be nice
to live in a place like that and so they want to read about those kind of because that's kind of solve
beacons have Armelagos concrete towers with a light bulb on top of there are also a kind of single or something like that I mean they the events to sentiment of things that are happening outside of and when something happens to those things and notification center which is actually an event so those are configured given its configuration file because we're actually interesting the means at this point and any system administrators in the room 1 of thus anything in this ring a bell something that is no notifications I notify maybe OK yeah I mean identify which is a 1st system monitoring that API to track changes of files and directories kind of looks like this so in fact bearings and identify can which is used to model that to monitor the changes to a certain kind of file a certain of in a given time interval and the you have at any time the result that coal-fired changes we now give an event and there's also other types because for example of process we can be monitoring whether or not a certain process we a process name we specifies running or not if it's not running and stuff to run we get event it's running and it stops an so and its friends there are actually several different types of memory-based consists of nodes that we're saying the works that really a lot and they're growing up scholarly views you can also write your own of course does not in the documentation here so you can check it out in our now this is where things are going to get a little bit more interesting yeah like that and we
will be nice if actually the reactor was like this the actually so what is this old
reactor as its name implies the main job of some reactors to react but not really in a JavaScript weight that we have to be Nusselt way
salty way another words and the reactor is the component there is responsible for triggering actions in response to about so now you see why we the event but of course or we need given system 1st but what is action these were sold and action is essentially the state but we define and what is actually going to happen in radiology something goes something like this something's going to happen right think right there's going to be an event maybe because the something was being monitored by become or something else and the balance is gonna be picked up by the reactor and the reactor is going to translate that events into an action for or actually state the reactors and defining their muscles configuration file into component of this old master engine of actually said director will be making these associations and the associations if you ever wanted and was you remember that he had attacked so the association is made yeah the time so we attack in the configuration file and we define which states that are going to cover that I think that's here is quite clear that you know that there's an asterisk we can use wildcards because some events or fire bomb or more than just 1 meaning and the media 19 attacks so for example this 1st point here is the event of I mean starting out or it 1 much all means starting up which has the 1 on the right place OK so this whole slide it's actually the main reason I'm here but it's the 1 thing they spend the most time while working with sold for here Schizophyta there with me there are a few caveats the state system that we just saw here you know those the states the inside area the state system actually limited and you can easily skip this where dilapidation and finance your reactor 6 and trying to write things that would normally work and the rest of salt the rest states that you have in mind not work here and you will find things were missing for starters for the reasons that are present here are not available in the reactor if you choose to use those you get an unexpected results also reactor states are actually processed sequentially the 1st rendered under that stands and to our work pool of it since the 1st process sequentially you're gonna want to me your states a simple and small as fast as possible so after long hours of fighting over the reactor and during the but would you care that's my head and this is the short version do not have the logic from your reactor states and this might be a bit too confusing because what's point and but I'm going to explain things a in more detail you should use your your reactors states for matching and then and decide which means to which states based on an and then just call your normal sold states that you have lying around and in the try to ask on logic here and you're gonna spend a very long and you won't be population so I I don't I don't if this section true that's what it looks like from the outside it appears there so this next because we're talking about 2 different engines even if it's on the same thing I like to think it's because of Python namespaces but I could be wrong but so to London and read the not logic so as was said now we reactor we events to states so if we have our testing events and we have our custom reactors state files the the idea is to keep it as simple as this and if you really have to do complex things and ensure that many many things are done when a given event is fired just put those inside the long public space so once we there was a reactor passes status and sends it to the work of Paul these will be 1 of the main sold namespace OK what can we use a reactor for them what example out of accepting all the keys of happening in Sierra Morena and you know this it's quite a hassle return you start I mean you know you have to go to master except the key and so on so forth and so as you might have guessed whenever I mean and tried to tries to authenticate an event is fired on Armenian finished starting out there is so for the purposes of this example we're going to assume that all means whose names start with a nice are going to have the keys follow except so um 1st of all In the 1st thing that's going to be doing when authentication first one to remove the keys coming from the unions that have failed to the next step is going to be to trigger I mean you start and now I know this is something this is just for the purposes of examples of every time I read as a stage in the middle of another language and other configuration management system by an out of it but this is just an example of 1 of you is have dominion re-authenticate during community so so reaching the uniform big states so if we are an independent status of of indication is that as and the name starts with nice we except the key as for the last stage when the mean and starting up this actually good practice that you can implement wanna Romanian finishes starting out we apply a high status to that meaning this is something nice to ensure that all your minions are consistent at least when starting now note here that we hardly we've been hard-coding the nights and maybe some other things run because as we said before we don't have range would have to be there you don't have if we can store information made available to the reader so today might wanna and our last component today that's going to be the API course so that has received the 1 of the main idea behind is to send commands 2 were running must and EPA supports both encryption and authentication different agation which is something that you might not see very shortly insults also has some sort of integration system the I of forward into the instant but against and it also has an access control in unit so know it's really outside the scope of this targets a very big thing to talk about but it is worth mentioning that the actually and the entire the things so they're managed by DAPI and controlled by another so if it's a REST API which of course use anything that can make HTTP requests and get information from course and information In this very short example we're making a request to a search and you 4 meanings and 4 pass the correct meaning maybe we're going to start getting better about that many In this case we a for the sake of simplicity were not using of integration now there several API and points available already on will result in they're pretty much self-explanatory but let me draw your attention to 1 particular theory natural this is especially for I exaggerate when the ventricle on the whole re something for existing is is that post requests they're done here will be generating events on the master side on the event and the post that was sent to it is going to become the best of our generated by an important thing because this is a special 1 point it's the only 1 that uh salt allows you to explicitly civil authentication in this particular part 1 thing if you're on the central authentication does not mean that you can do whatever you like uh you're expected to implement some kind of security why would you be single authentication well I like to think of uh apps that can barely perform and HTTP requests that can better understand the URL so they can only to request without special hardware token that is specified so that's what happened no prob out from all the rush that which has been in the whole world about we put them all
together the nice friends and no 1 of you might be laughter here you've seen a lot of information and ideas thing that might be a little bit confused and they show you can do pretty interesting stuff we'll
always all the events becomes the reactor and the not more graphical understanding of what all this connects to the we 1st the beacons and the API the main interesting point about this to is that they're related to elements outside so you know becomes wider things outside sold and the API the so any might make a request so they're both related to limits else now both of these 2 will be generating events in our event system thing event those events can be later picked up by the reactor even when we define inside the reactor which then can be translated into solid state not with the greatest possibility of having to manage your entire demo slash slash workflow infrastructure cancer great power and there's a liberates reordering of phrase here because if you configure so probably going to have full control of everything in infrastructure in the world of everything I'm from within itself so what such you're expected to know what you're doing and you should always rely on a sensible way of doing things for example be aware of the security risks you might be tempted to you know the way too much power to solve and that's a good thing but the where somebody trying to the I'm not leaving so to finish this off let's take a minute to go at all you can do that with all of this are just being a new a couple of examples from the top of my head and in the view to thing to rest that's because that was sold to kind like a batteries included approach to give you that the space to create your own solutions much like Python which is while of so just name example the set of also human anybody knows what self-healing with consists of anybody here this service OK so a more humane words save self-healing is the ability that we keep erotic nations or systems the ability to repair themselves whenever something bad has happened whenever they encounter an adverse situation and on the wrong that's the thing that's why it's called so you know all this might just be a REST API call away because it's in your application you can identify the bad thing that's happened can be corrected by by something that can be automated you can do it we API article because sold can have control other example and I think that many of you have encountered this let's say half your team refuses to use Jenkins on CH 2 when you're using if you're not because you can even be wherever they are using and integrate the rest of the Bush Field test deploy let's see I cycle with some you can manage it with salt of another example if you were talking about scaling both upward downward sideways growing shrinking you can prepare for it which sold uh you can also trust sold to do some provisions we have included here about so little has sold cloud the supervision colleagues and last but not least with a good to because of that you can make sure that your environment are consistent you know if you have things that supposed to change and you suspect that somebody tends to do nasty things with the beacons you can react immediately upon any changes that you're being and want and so these are mainly all the examples that I could think of with the short and that was given before um I really do hope that you can leverage what you saw here to come up with your own solutions because I'm sure that your problems might be worse than what I think is presented in half so as the dots and as for last year all the documentation is in the official so that the mutation and I really encourage you to the take it to a degree I could have any particular questions there's also the possibility of water and the guys of the salt free no channel ICT I do not and we're reaching the end so now we have time for some questions feel free to should and can you come compare salt with and support the I'm going to be honest with you I haven't used sensible I know that it's my maybe as a more basic approach what I've been told from the people that have tried both is that and the like some components of salt I like the reactor for example so goes from those sites the but I was wondering how 1 can use the computers sort deployment or is it the 1st deployed in the workstation with it or always just which fits to set up the system and then you will need to refer to a proper determined to attention for the application deployment by yet workstation so so that is to say something but some of this data in deployed time what search terms the web 17 that yeah and naming them maybe that was covered in the previous which is more basic but none of that's a very good question and yet you can do it you might have to and what slightly more manual approach in order to generate for your environment and you can certainly 1 thing if you're thinking of doing some very model provision and you can also do that and I know that exactly we sold points song but sold as an integration performance provisioning software was mainly written for public but not as salt integration so you can do the whole cycle from I Hill and I understood that the communication channels is this is correct the has some way of working with same but it's not the main way that know what I want to ask is how you can do and I mean you've set running Windows etc. good question actually and it is possible that I'm sorry that's I have been at to be playing around with that that find a way to insert this such data Windows using some wings but has a has a search it seemed to be working in dumps corners of the output options yeah I mean as long as you you know aware of any limitations that you might have an issue for the rest of the system the shares same thanks for the talk i think of using pixel for 3 months so really cool others also want to stop er engines reactants some some older children and currently under looking for the way to makeup lost and I want people to get information from the Council or from let's see the during the blood to get some keep your information from Cuban storage external is it possible I will have to get out I'm not entirely sure about everything appears to be extensible insults so once you were not yeah maybe this but how do you operates hold without this stage Forrester any good approach to do the the upgrading sold will station in the master and the I and I'll add think and this is where would you were going with the question he's only all you need you need to have access to the system you need to have access to to it for the new version and you also have to restart creation no that's actually that's 1 thing that it's not still not found very well and salt is restored unions whatever there's not really because you can't do it from inside mosque because they're going to be loosen communication so yeah it's a question of what tricky spots still the the questions about the endemic I think it's possible as old as a mechanism to and we can get to the left from external so you can implement quite model to there is a quite cold less in which is to be made to a more reasonable and my answer questions how do you test your states with us with has several breakages cement production due to human error I know I know uh I don't have time like that production system I mean a useful for personal use these and the luxury of working with the salt environments but I know where you go with it so that it would be nice if you could have a development environment or to a to try things because there was you made a change to to the the States so there's like if you will walk it's going to you have to keep them together logs and be very careful what you changed your bound to have the last change you may pose a problem if you see a problem that is In work and how they will handle provision in new servers and how do you handle your inventory all servers yeah From the point from the located so 1st question provisioning is done in 2 parts if you start with an overview of provisioning you have to use something like form and allows you to do this system and then applies all states to so sold this knowledge is like copying the way in and it's not like cancer but way it's it's not that is that doesn't have the ability to provision system from their mental the what this system is already installed and has a million running you can do whatever you want with and ask for the catalog the inventory from the perspective of a master all the Masters sees Armenians so it's up to you to group them using node groups or grains or whatever you you seem to be necessary it would be basically setting the and your own you groups of certain range uncertain minions to identify them from the perspective of the master but essentially there's no know where that distinction In fact I when we talk about this in the note so the made master the master of masters the full meaning of connected to it even those from the lower levels in the so this is in response to the question about testing results we week we do this we use will use data on your local machines with a bachelor's millions of and then spent a number of the atoms and actually touched the states these to some minimal so that we can catch human error like that we have same problem we deploy across hundreds of machines simultaneously in 1 area could really message has a like I've tried with their locally as it works pretty well but you can still different kinds of yams we use previous to Europe going to resent us and similar all environments easily interest thank you the and is and high things thank you and I'm seeing bottom here I'm seeing that most of the questions we are asking about salt with the this that South can't can be done perfectly using unstable like the shallow system configuration all so maybe someone asked about service management on and I I know you said you haven't used on small but have you had about someone using unsealed sold together and no pretty much when somebody chooses duration models they like to stick to to it i has to to do with the learning curve and a lot of stuff so it will be harder from that very few things that seem promising it's quite different results in the both written by from and even the old he's younger there are quite different so no I urbanization wants to choose 1 technology and stick to it are the well that's interesting I mean the sizable can be used for the nominal provision parts and maybe for the rest or for the reactors that you could simply means those and if you will good just to rule create future machine to install full pastor of and it to to configure it to the Toeplitz state this I would think of as the fabric Python modules is like the 1 that that comes from Chronicle the amount of users then right is stage of true through basic provisioning strict rules started tradition to work for clustering and then use the full posterior and the of salt yeah their you can we have time for 1 more question for much he was thinking


  607 ms - page object


AV-Portal 3.9.2 (c7d7a940c57b22d0bc6d7f70d6f13fde2ef2d4b8)