Using systemd to high level languages

Video in TIB AV-Portal: Using systemd to high level languages

Formal Metadata

Title
Using systemd to high level languages
Title of Series
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
2018
Language
English

Content Metadata

Subject Area
Abstract
There is so much more than you can so, than just starting and stooping your service, when you start to interact with systemd from within your application. Lets find out!!! We'll explore how can you interact with systemd from your aplication, by describing how pystemd, a python binding for systemd, was created at facebook, and what type things your programs can take advantage if you just interact with systemd more. From executing section of your code securely to create development environment on the fly, There is so much more than you can so, than just starting and stooping your service, when you start to interact with systemd from within your application.
Loading...
Facebook System programming High-level programming language Energy level Spacetime Physical system Formal language Library (computing)
Demon Group action Presentation of a group Service (economics) Decision theory Execution unit Virtual machine Function (mathematics) Mereology Field (computer science) Facebook Goodness of fit Web service String (computer science) Bus (computing) Integer Information Codierung <Programmierung> Message passing Physical system Scripting language Information Decision theory Software developer Interface (computing) High-level programming language Bit Instance (computer science) Group action Software maintenance System call Type theory Data management Web service System programming Bus (computing) Quicksort Physical system Library (computing)
Domain name Web service System programming Physical system Reverse engineering
Functional (mathematics) Service (economics) Interface (computing) Multiplication sign Execution unit Instance (computer science) Parameter (computer programming) Symbol table Attribute grammar Category of being Web service Word Network topology Source code Object (grammar) Endliche Modelltheorie Physical system
Identifiability Information Interface (computing) Electronic mailing list Theory Login Parameter (computer programming) Theory Category of being Type theory Object (grammar) System programming Representation (politics) Object (grammar) Communications protocol Representation (politics) Address space Library (computing) Physical system
Keyboard shortcut Implementation Observational study Direction (geometry) Multiplication sign Software developer Cohen's kappa Parameter (computer programming) Type theory Object (grammar) Bus (computing) Electronic visual display Category of being Library (computing)
Web page Keyboard shortcut Implementation Inheritance (object-oriented programming) Multiplication sign Mereology Formal language Facebook Latent heat Mathematics Computer configuration Software Authorization Bus (computing) Implementation Library (computing) Physical system Scripting language Software developer Keyboard shortcut Arithmetic mean Word Wiki Exterior algebra Lie group Blog System programming Communications protocol Library (computing)
Information Code Ferry Corsten Multiplication sign Execution unit Sound effect Parameter (computer programming) Virtual machine Formal language Revision control Propagator Error message Social class Pattern language Physical system Social class Library (computing) Exception handling
Point (geometry) Code Ferry Corsten Multiplication sign View (database) Bit Cartesian coordinate system System call Formal language Different (Kate Ryan album) Ranking Social class Object (grammar) Error message Writing Exception handling Social class
System programming Videoconferencing Ranking
Service (economics) Principal ideal domain Code Decision theory Multiplication sign Execution unit Limit (category theory) Function (mathematics) Solid geometry Mereology Formal language Local Group Leak Goodness of fit Network socket Bus (computing) Energy level Circle Integer Physical system Installation art Email Demon Information Key (cryptography) File format Software developer Interface (computing) Root Process (computing) Personal digital assistant Network socket Radio-frequency identification System programming
Web page Group action Inheritance (object-oriented programming) Principal ideal domain Code Codierung <Programmierung> Execution unit Sheaf (mathematics) Set (mathematics) Barrelled space Perspective (visual) Type theory Energy level Physical system Execution unit Inheritance (object-oriented programming) Information Code System call Type theory Process (computing) Network socket Sheaf (mathematics) System programming Right angle
Context awareness Asynchronous Transfer Mode Group action Distribution (mathematics) Information File format Code Software developer Multiplication sign Cartesian coordinate system Portable communications device Web service Process (computing) Cache (computing) Integrated development environment Military operation Gastropod shell Website Virtual reality Physical system
Asynchronous Transfer Mode Context awareness Code 1 (number) Set (mathematics) Temporal logic Binary file Product (business) Mathematics Cache (computing) Military operation File system Ranking Website Physical system Context awareness Distribution (mathematics) Software developer Instance (computer science) Cartesian coordinate system Uniform resource locator Integrated development environment Configuration space Gastropod shell Virtual reality Library (computing)
Scripting language Context awareness Execution unit Context awareness Scheduling (computing) Stapeldatei Ferry Corsten Block (periodic table) Multiplication sign Software developer Virtual machine Database Database Instance (computer science) Process (computing) Gastropod shell Process (computing) Data structure Scheduling (computing) Backup
Scripting language Area Execution unit Service (economics) Computer file Multiplication sign Execution unit Database Bit Limit (category theory) System call Attribute grammar Data management Process (computing) Gastropod shell Ranking Process (computing) Scheduling (computing) Backup
Scripting language Backup Mathematics Demon Military operation Multiplication sign Execution unit Database Information Process (computing) Electronic visual display
User interface Scripting language Execution unit Scheduling (computing) Building Backup Service (economics) Multiplication sign Interface (computing) Execution unit Formal language Mathematics Web service System programming Bus (computing) Process (computing) Information Remote procedure call Quicksort Electronic visual display Physical system Library (computing) Physical system
Slide rule Email Functional (mathematics) Service (economics) Multiplication sign Interface (computing) Parameter (computer programming) Type theory Data management Loop (music) Computer configuration Bridging (networking) Operator (mathematics) Netzwerkverwaltung System programming Cuboid Object (grammar) Physical system Library (computing)
Service (economics) Execution unit System programming Object (grammar)
Keyboard shortcut Inheritance (object-oriented programming) Interface (computing) Login Cohen's kappa Login Proper map Attribute grammar Error message Object (grammar) Bus (computing) Quicksort Object (grammar) Electronic visual display Category of being
Inheritance (object-oriented programming) Interface (computing) Netzwerkverwaltung System programming Object (grammar) System call Attribute grammar Social class
Data management Logic Multiplication sign Quadrilateral System programming Virtual machine Object (grammar) Instance (computer science) Login Local ring Exception handling Social class
Facebook System programming
okay cool so hi guys
another lever approach engineer at Facebook and Instagram and I'm here to talk to you about how about we use system D in high-level languages system DS already written in C so that's already a high level language but I'm talking like higher level languages like a Python know the API sorry null ya know JavaScript PHP all those languages that are not really like nuance and stuff okay cool so um basically this this talk will have like to two portions one is how we as Facebook the other library
that talks to system D directly instead of shelling out and then a few fun things that we do with it they leave this presentation is not to say hey guys go and use our library is more on the sense like hey like this is all the things that we learn by creating this library you should all go and create your own libraries in your own languages and if you also want to do one in Python that's the one that I'm going to present and go for it like this is really fun and this is something that we can all learn from so let me start with the initial journey on how we start with this so um this started basically when we were at Instagram we needed to migrate all of our fleet to CentOS 7 because that was part of the of what Facebook was doing Davida has talked about this for two years now he promised this that we he promised in this year that we have already done this and we were one of the Hellas groups to do this and the reason why it was hard for us is because we had a lot of maintenance Script that rely on the underneath a service manager and one of the things that we typically do is that we would execute stuff like et CET Cassandra status and then we parse the output and then that's how we decided if Cassandra was running or not running and all the things and so all the things that were not really know it's not that we're not possible but there was not a good idea to do it with system D and for good reasons a so for instance system D comes with systemctl status which is really great for humans but it's really hard to parse the output and system D does provide like a lot of to do this from the command line like even systemctl has such old status that will kind of ask you like individual fields you can also use bus ETL and all those stuff but if you are in a high-level language again I use this at like a synonym for Python to do this you have to chill out and at least for my
personal experience telling out I've considered bit a little bit clowning on those things so it would be really nice to have like a really nice and native interface to talk to system the yeah so it basically all starts with diva's right so from an external developer not at just on the developer but from an external developers that want to interact with system D this is for me the best decision that could have ever gone into system D like using diva's as a way to interface to interface with system D this basically allows external services to communicate through this demon and and get all the informations that you need from it in a way that is not as clowny or cocky as like executing a command getting a text out parsing the text and then if that text is supposed to be like a peeled convert that into string sorry convert that into an integer that is causing some problems so it was released kind of a surprise to me when I start working with this that this is also how system the most internal tools work and it's kind of because we all know that divas the Divas daemon is not present at every stage of the of the of the machine like it's not really person at the beginning and at the end shut down and start but for most general things this is how you communicate a difficult even system digitally communicate so first like if you go and read system decode which I'm assuming that everybody has done it you will like see all this type of calls like systemctl a communicate to the manager a sitting the run which is something that I really like to use and abuse also communicate survivors to start the transient units and even match in CTL like instead of telling all the changing names face sort of we know things that you could do just communicate to the Divas daemon running inside I'm a teen I'm pretty sure that I'm going to get corrected about this but at least this is how I view it from the outside so the u.s. is like really great and it's something that that we could really use in general like if you forget all about this and you need a really good IPC layer to communicate like you see us it's really reliable I will spite people can tell you otherwise so just everything start with divas like way if you're going to interact with systemd from a high level language you should literally just start with divas so let's see a little bit how give of things work
so system D comes with a nice tool there's Basset al but CTL when you
execute it without any comment it will just show you all the services that are running on your system and these are all things that you already know since like system d1 or hostname that is called the reverse domain something I'm sorry I've blacked out of that well basically these are all the services right if you go and explore one
of these services you will see objects and objects you can always look at this like name of instantiated attribute associated objects on on the system for instance for system D these are all the objects that you can connect and then you see even though they have like we're names they all in Target or service so if you don't know anything hope this thing works but you have only work with system doodles doing like star creating you and it's stuff like that you will start to see like there's a certain for symbols and this looks like really programmatic way to access this this data um so if you go and actually
do an introspect into these you will see that it has interfaces that has names that are really similar to what you will expect them to have like if I'm expecting if I'm if I'm inspecting the service a service unit you will expect I will have ordered three desktop the system did service and then inside those you have methods and properties and a pretty sure that this is really known for everybody but at least when I started this this was all new so I think it's at least worth mentioning a method it's just a fancy way to say functions is something that you will call and will return are something that you can give arguments to it and properties it's again it's another fancy word for just saying a property is something an attribute that your object has and did contain three times I always think of properties like functions that you call without arguments even though that is not true but that's a really mental model that I have so the thing that make a diva's a diva's object that you actually that makes this really
powerful are the three interfaces that are at the beginning the first interface is the one that we're going to start off using and it was the one that make possible to actually create a library was the introspective all that allows you to without knowing anything except the address on the object this will give you the all information that you have to reconstruct the object and actually interface with it peer and properties are things that are there and are really cool that we are not going to touch really much but you can also use it so introspect introspect is a method of
that interface that if you call it you will get an XML representation of the object and once you have an XML representation of the object anyone who has lived through the 90s knows that you can construct anything right because so did it so why can we do it with diva's so in theory that's all you need to do to work with it so how do you call it I'm getting ahead of myself because I'm using system D library there's nothing in my talk so far that talks all system D I'm just talking about diva's but I'm getting ahead so basically you call that method and then you will get something like this that is the XML and you see that it gives me interfaces a list of interfaces inside you have properties if that has the type that it will return it has the type of a is a real name that you will use to a unique identifier for the property and then you have methods and methods can not take arguments or they can take arguments and then you
have arguments with a the type and the basically the direction if they're in its arguments that you give you also have like the return type studies eventually they find summer and so some implementation of of Tevas also returns the name of the argument that is kind of useful I mean I would prefer that it were there but if it's not there it's not a big deal so we now have a way to construct things so when we were going to start
building this library we had to find like underneath libraries that we wanted to use again I'm a Python developer if you haven't noticed by the hundreds times that I say Python but so I wanted
to use Tevas Python because it has Python in it and it also has divas it makes sense the problem is that that is a binding to live divas that is really old that is basically written exactly like a month later after the implementation of a specification of divas was written so it's it doesn't have all the experience that we have now that we have this protocol is pretty mature and also the main author of this this library said a disclaimer that says the US python might not be the best he was finding for you to use so if they also said on use it then thank you I will move on okay the next one was five Eva's which also is based on key divas and it was really it was a really good alternative I wanted to use but with that said I spent a few times reading learners blog and I find a steep bus as part of the system the implementation um and this was actually the best thing that we wanted to use so
we wanted to create a library to win to systemd not to link to the two divas divas was the protocol and you think this thing will basically give us the same way that system D itself means to leave us and do everything and also it gives you a little way of cheat in the sense that I could just go to system D and read the code and understand how this do it and then I could implement it on my script in Python and say hey I create something but no I didn't mmm for your language if you want to do it there's this page over here there's some free desktop that has all the bindings for many languages you should go and again write this in groovy have fun okay so the correct things that okay so we want to use as divas there's no Python library for SD bus in fact there's no paper lever to vim to live system D I mean this is not there is not there's few libraries and none of them did whatever we want so all developers do when they can't find something they just build in themself so we at Facebook creative piste MD be some discipline of words because it sounds like system D but also start with PI as every PI for library should so yeah and I mean think this hard this was easy great other options were peeled uno which basically speed one but in Spanish so yeah we didn't yeah okay so a yeah so basically we did it because we wanted to pin to SD bus but we also want math another word meaning to system D we could be in two other libraries that were present like STV Mon and do all the things that Terry that talk of his talk that was really cool so how does this work a we
basically Python this library connects two divas get the information and you generate classes nothing far of the ordinary I'm pretty sure that everybody can come out in five minutes with code version of what this does but here's the
fun thing that you learn while you're translating basically something that was written in C into Python is that there are seasons or Python isms that you need to translate from one language to another first is in C there is no propagation of exception so you see this pattern all the time when you read code you make you call to a method that returns exit status and their side effects on the argument and then you need to check for the return status to get to get a unit sir you need to check for the exit status to realize if you need to fail or continue and then if this method is returned by another method then that methyl also needs to check and then there's this propagation that you have to explicitly do in Python and other languages where you look up exception propagation you all you need
to do is like check the exit status of the first time that you call the command that actually has this behavior and then you race and when when you race the exception that will bubble up everywhere else so you don't if you want to stop when something fails you don't do anything you just let the application or the bit 1bm it does that for you if you do want to stop so if you don't want to stop then you just try accept your errors so that's one cool thing that you get and then you're gonna see those difference if you ever look to my code that I brace a lot instead of chicken and then I try except Allah and the other thing is that a systemd at
least from my point of view it's really written in a way that it looks like it supposed to be like object oriented but it's not because it's not so usually give like this struck at the beginning of almost all calls so this really lends to when you are translating this into a language that is object oriented like heavily overcurrent heavily object oriented you get into those things so you have the class estabas that should be like then you delete the first name and then you just a dream all the methods and then they call self Devi of doing this is that if somebody is writ is used to write Python this comes really natural and it's something is used to write C and wants to write these
things but in Python because somebody asking - it doesn't have to learn sudden so much new things it just need to learn that it needs to replace s diva's underscore we s diverse and then a dot that's not that big video so now how
this look it's basically you install as everything with peepin stop listening then you call it like that and it looks
very simple like if you do org dot free desktop to system d1 you will install solid sorry college from piston little python and then you call the unit and then you start the unit in this really bitonic way where you don't have any information about how it is and this will construct everything and then you can start and stop it and this thing over here encodes the idea of the interface that is completely evaluated when you call for the first time unit yeah now this is again this is you can do it you could like if you don't see you can even do it and see there's nothing magic about this the great thing is that now for a Python developer or or or these developers that are high level em that they have never touched C and they also don't want to see you give them like a really nice interface to do stuff that they first didn't knew how to do or that they will chill out and then do a really hockey and let's face it clowny way to to get their output so these things are cool also once you call this method that it basically automatically calculates the thing that it is to replace put it on the bus everything you also get the out there real a return value of the thing that it will in this case would be a job so then you can verify like the job through the US and everything right tight nicely and then you don't have to do any experimentation I say that a really thing I said that I really think that system key was a really good at making the decision of running divas the other thing that I really think system did does really good is giving you output or information about your service and I'm preaching to the choir probably but a but when you loo systemctl status you get all this information and sometimes you actually want this to to do in your code so you do the same thing and I guess the main pit and again they may impede you you get it as a integer because this thing is typing but then with piston D you formations okay so with twisting D you can do a you can actually like do socket activation I mean you don't do it to a second situation but you read the information from from circle activation when you just get the same headers that you would get by paying them it's a good system D and you'll also get from of T and then now a Python developer and maybe I'm selling them short but hey that is used to the like do socket socket and then socket at least and then so gondola SEP or pine now in his own language without changing anything he can do these things so that's the first part and now let's talk about fun things that we can do and a little disclaimer everything that I'm
gonna show you you can already do it with like systems Iran and then bad scrape and everything I'm just showing you how you can do things that may not be the best way to do them but again you
can do it from a high level perspective so the first thing that I really like
when I start working with system D and the things I really wanted to have this idea that I could decorate a method like first in this ping over here that it could be like even like a normal call and then I wanted to decorate and then just run this section of the code in a different secret or even in recent process so I actually went on and the other few things this will all be available on the Beast and D page so it's a promissory right now they are not because I read only for the stocks so my idea was that I could decorate this method and then make this method run without any network access a daily of this that sometimes you have to run or like non-secure code or code that users provide so it would be a good idea to just make this thing running on different Seagram and again this is not the best way to do it but it's a at least interesting way of doing it so yeah so this is how I solve it I decorate that and then what that will make it will start a unit that it just be a really big no op it has to remain after X true I really know that you don't need to do this what I just put it up as an example and so remaineth directly true so this will execute this will not do anything and they will stay there and then you set the type one shot you set the type delegate to true and then you said they serve the C group settings that you actually want to preserve and then you start your settings your your your method to start your method I do a little trick that is the following in a main process I start this as a child I fork the great thing about doing the fork is that I can still hold all the status all all the barrels and not information that I have fun going process so I have like if I've declared a global variable it's available to this method but I start I wait for this thing to actually start to have a peep and then it call this little great thing that's called attach process to unit that basically will allow me to make this process move them to this C group or to this other process yeah assume that I said something that makes it so M so what so then this is what happened in the parent and then in the children I start the process I wait for actually to change the unit and to be running on a different unit and then I call the methods what this will do is two things the first thing is
that I can call this and then when I try to ping it will not allow me to think even though my application does have this formation by me changing into a different see group it works this is not the best way to do it but it really it really works good for this the other thing is that now since this work in the different process you can literally just do systemctl status the other process and then you will see this running and then this give you all all the informations that you want so yeah so this was one one cool thing that we did and I think okay now I
am I'm so when I still have time so other things that I have is that a just a raise of hand because how many people here are Python developers or code in Python I love you all guys okay so um okay so I'm not going to preach the choir in Python you have this idea of virtual environments right which allows you to basically set similar but different environment that you have on your system with Python so you can imagine that there's an analogous concept to the idea of portable services but having portable Python environment so instead of you just ship in like a whole system like a whole distribution
thank you like a whole distribution you just cheap like your Python code you just ship the Python executable and the live Python and then through the magic of just setting up stuff you can make it run in a way that is completely isolated from the Python that you have and this is one example of how we do this but for instance you can just start a temporal file system that's awesome and then you just bin mount all the Python that you bring from your application you just mount it into this environment so you
can mount the the literally the binary and then you can mount whatever libraries you want to preserve a then you do a few configurations and you run and then it is a good way to actually create virtual environments on the fly in the system the in a system D context instead of just to invert or M so this was actually really cool yeah with this said you can do the same thing it doesn't have to be Python one other thing that we have try is that actually instead of doing this is bash so we can actually make a when I developer for instance who wants to try like they have their own settings that looks really similar to production but they actually want to try a change interchanging libraries with the ones that they are developing we change this to bash and then the mounts just change the location where the lever should go with developer one and it works really well the good
thing is that this work in the context of the process and then when you exit the process it goes out and again the magic here is that you create PTY something you've been siesta Venus TD out this is the last example that I will give and I think what is really running out of time and I think I really want to have questions at the end so one typical thing that we usually do as as a for
structure people is that we need to for instance we have to the Commission a whole rack of machines so for that if these are data bases you you should need to drain them then take the data put in somewhere else and run it and the first thing that you do this you create a batch script you SSH from a scheduler you execute the batch script but then that batch blocks so you block the thing and then you don't you cannot actually run more than a few because you don't have resources to run them so the first thing that developers do and I've been guilty of these is that they say oh so
I'm gonna create a service that is going to be listening and then it's when he gets the call I'm gonna so process something creative and then this is gonna start working and then I'm gonna monitoring what is doing I'm going to finish I'm gonna call back the main process and every time that I had that it's like that's a service manager why would you code that if you already are running in Linux with a service manager so a good way to do that is that instead of doing that just right there same
shell script that you have though a little bit know a little more interesting stuff but literally just run it as a transient unit so like this could be like a normal unit and then you just start it but if you want to create this file on the fly and do lot of stuff you could literally just create a service areas are transient unit you give all the attributes that you want you can put a you can put a limitation into your script change the user and the good thing is that you should always try to use notify when it's possible the reason why well so this is a non-blocking and then you just return the name of the unit that you created and then you just monitor that unit now
how do you monitor that unit one thing that you could do and it's not really useful but then you could use like notify so notify you say ready equals one if you like the status but what if you can put text into it and you can like you look like none over I've use this but a like in your in your
backup script you go through your backups and then every time that you make a change you change the status if you do that then if you're one with you
monitoring your service remote tools you will always see the change that you deal then you can actually get it from the text but that means that your scheduler
what it can do it can always kind of go
and connect to that particular unit and then retrieve the status text and this is useful when you create interfaces like like human interfaces like you allow the script that is doing the backup to communicate how it's doing the things that way you can read it and that's mainly it mmm I will not touch that one ask me later if you want to vote that final thing is that peeling - based on the sort or bending - live system this one you should all do it final language doesn't have a bending do it yourself and that's it so thank you and questions [Applause] I have one minute so at least one question or none go for it so I have to question the first one is you showed before there is some auto generated code out of the D bus introspection stuff when is that happening like at Build time Jo so in in in my library you can do
whatever you want you're free to write whatever you want it's here it's like when this with you create the object and
this within Python world its execute a method that's called enter and that method enter you get go to system go to the library sorry go to system the go to vivaz called the introspect method gets xml construct the object and then you return the object that you constructed the only
good thing is that these interfaces are really stable and you only need to do this once we don't do it we do it every time that you create an object but you only can only need to do that one okay and the second quiz was like if I'm not wrong there are a few function library function that are asynchronous already insistently oh yeah I'll do you and that one is like his own haven't loop I think it's like how do you bridge with the Python the sinker and stuff so i don't but i we off so yes that that's one for the first thing that I create this library I created with the specific intention that Python 2 has to be supported which everybody in the Python world hate me for um I'm Python 2000 support I think I owe like out of the box so now that we're moving into Python 3 world it's like I will do I think are you and I will make this library completely a thing if you want you like it's not cool you're welcome cool gentleman in the front that I don't very have never seen on that slide right there what is replace oh so um that is basically like the start and the stop method they this is like sitting the prove yourself so then I can't give you the better answers than that but they take arguments on how you want to start like how do you want the manager to actually start or stop your service the most evil one is replace go ahead it's basically if you start a service and this contradicts with another operation that has been queued before what shall happen she'll the new operation replace the older ones or she'll the you can fail or something so I I want I want before you if you have asked me for you ask me I want to say like this actually was one of the coolest thing about working in this type of library is that you get to learn your own quote how these things actually work how this so such is mail because you get to like I call this method doesn't have arguments like what and then you say like oh there's all this option there's like three options I believe what my question was regarding like if you could yeah and then this slide here like you're saying that the introspection happens at the time the the object is allocated right yeah so this works for any any device service yes let's say talk to I don't know network manager oh yeah so so it I don't have example for this let me see if I can get this one
yeah so basically this is how the my work you like I have sorry okay so this
one it's like a custom object made
understand that is the concept of unit but I have an object where where this inherit from that basically allows you to specify any any service and any object when you read that object and you
see the XML I understand that everything
there like if it is ordered free login that one I understand that every every interface that start with that it's like an interface proper of the object so I exposed it like this thing as a proper attribute so then you have like your object that you created that you need sort of user would be this one and then you will see it like it works like that and then the method works and the other one that has different names they are
not exposed to the separate attributes but you can always like get interface and then give their proper name and then if they will have so what's the work then then let's say if I want to talk to network manager any other divas interface what would I do i would first
subclass this generic object saying we feel like a handful in of things and then that's it right yes exactly but in general like you don't even have to subtract it because the base class like it's work on its own you have two classes like to make like future calls not necessarily to actually give the the name but you will literally
just if you want to talk for instance I do this a lot like with machines with Mattoon one you give this one and then you give the object that will probably be either the machine manager or the machine or the machine itself and then
like you say everything works like Rare of the books except for stuff like time date which are like really small and
they not really confirmed this logic a we're gonna talk about those later but but for things like like mateesah TL or login which are like really big classes and they like work exactly exist only one perfect sounds great I don't know if I have time for more questions I don't have time for more question but [Applause]
Loading...
Feedback

Timings

  574 ms - page object

Version

AV-Portal 3.21.3 (19e43a18c8aa08bcbdf3e35b975c18acb737c630)
hidden