Bestand wählen

Scripting across hosts with Chopsticks

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Erkannte Entitäten
so I'm here to tell you about chopsticks which is a new library uh I wrote of over the past year for some
running Python code on remote hosts them and Daniel part of verbs is this
talking to Daniel some of them where I can add a k through them watch optics is how to use it and then take a look at some the tricks to make it work so some jobs is fundamentally
allows you to run python code on remote politics machines and in parallel SSH with nothing installed except the estates of our them and and Python system like the I so it behaves a lot like I have been calling it multiprocessing age but the simplest and they can also do some things with darker but so will explore some the parallels between how I'm using SSH and Haiti's docker CD and other things this is a chopsticks script
that so to find importing a help my get management functions so there is a there is a get revision function somewhere in the code base but there is this group object um and I pass a list of strings which I bought a so if you if you just pass strings then they all host SSH 2 and then we can call that function on will host and then the
results I'm and I would like to show you the the kind of implementation of the function so that's how you might like to get religion it's just surprised and and you could write
a better job so there are some restrictions of the
cut to the code the can run a remote hosts a has to be pure Python and the function
premises optical so you you when you look the running cool as they only the callable this past the remote site um and the return value must
be Jason serializable and that the reason for that was the kind of security concerns that it may be your of your your so they get compromised you don't want on your laptop running these orchestration functions to be compromised um and pickle is also so let's look at some of things we can build the tunnels so got holds for SSH from and docket so
these are the symbolic docker on communities like spin up the docket container and by default it will be a a DOC a container that is destroyed when the way in this case the context is less and so so he considering users context I think in my 1st example was using as context managers the context manager closes the connection I'm at the end context and of course that means that the any remote
state is torn down for a while the connections these of course you get a Python interpreter another and that could be doing in the B to the spinning out some of demons for doing networking um C 2 runs of fruit was another user and local is is just spin out a Python process from which the price the least useful of these some and if you
passed in any of these tunnels to group then you can run colors on the unit in the tunnels instead of SSH and so this an example of
using a group to execute code on laughter differ Python versions and say for this you need to know nothing to guess what topic chopsticks and docket and obviously dockable fetch the images and and run the yes not all of them like kx type that and um
so if is there that when you're doing this of course uh you are running is a few if you're doing this specific example where we go Python 2 7 and Python 3 of the code that you're reporting process because compatible um it's a bit of fudge trying to cool codes from as if you calling Python to you and pickling objects from Python 3 and then you can expect to see a strange behavior when you that
was read somewhere and degree Python 2 to Python 3 then you expect to see stuff in Python 3 that is bites and and that's the that's so that's just the nature of pickling and so maybe this is not without pitfalls the groups as well have the ability to do but they they act as sets so you the you crates and and these are not connected at the
moment you crank them so this is just sort of be all hosts operation here is offline and then the tunnel is automatically connected to all of those hosts to execute is inventory what's the function so you can see you could use you could find in 1 module of hosts and then you could
dynamically select some of those ties to run operations on and a massive and implementation is event a subgradient the but
it's mostly so does look up all the operations
that the tunnel supports this will be a can of wax on wax off the chopsticks some say every tall supports these operations and like explicit connect i which if you run a metallic will raise exceptions so that tunnel will always raise exceptions when the thing fails groups tend to that so that you can get a successful bits and the partition the success
of its in the error its and during error handling so you put infection particular exists to support string of large files um that's because you would want to pick all the entire thing and store in memory while it's being transferred lines and and the
context manager is explicitly shutting down the tunnel that's so that's an example of using fetched affection follow from a remote host to affecting parts of also for being president would do that but um the fetch for a group of inter-clan hasn't special behavior where you don't overwrite the same file for for all of those hosts so it's a key by the name of the host construct a unique path to write this into and apart from that they behave exactly the same on a
group has on April and also make them as the race failures which is just a quick way of handling exceptions for a group so so far the operations we've seen
synchronous and so if you make the method call and that it will block until a for a group of block until all the heist a Catholic their result and sent it back and it will return you the the response objects which has all the responses from all the host and
to support the parallelism in a group chopsticks is a under the hood um so to there is also an API to make use of that which school q which is my attempts to to cry
at asynchronous API to so that deal effectively with the asynchronous less so in interval parations you might as well to speed things up and so do you can queue up at the top of operations and say are of an so in this example um you can compare
the well so you know we're we're putting a different operations on the individual finals week we can send individual operation so rather than a group always the same parameters to the function for all hosts the whereas in the in the 1st example here but we can pass different parameters to to different hosts and then
run the all in parallel and all of those operations that we return a and a sink result and you can attach backs and and so on In this 2nd
example you or white passing a group is the thing that you calling the operation on it would as to what they think about parallel so that actually sending each of 3 false each of 3 hosts the um and if you can imagine how that affects performance from the
but because there's so the variability and actually could be a very wide variability of listening to different networks and then apply to the switching of all of the the gaps the version with the that you can complete foster
economic the example a a bit clearer but you can notice in in terms of the time the 2 example finishes of it sooner than the operation with group and as you add more operations and small variability that would go a lot faster some the OK so my damage time
chopsticks also work straight out of a Egypt tonight but so the yeah I think that's and tends to be really hard so OK
so conference or a meeting and I have a I have a connector tunnel to assist has just 1 with others because of and that's a story that
hasn't collected so it's defined loosely so we'll see if it connects 1 at the core functions of functions defined in the cell again so that's number processes running on that's the and say to make that work there is some special support for pickling
um so serializing Cobles find in under on the main um and a president to care about that unless you know what it means but that is actually sent as source
not as as it normally be imports work
by by request this this module will send me the code for it the and it uses these we impose system to find the in this case it's sending the source and anything with a real relies on so it knows that relies on last the I sorry so you um yes my 1st say like this all works with the standards SSH binaries so I could to get my SSH binaries to you have a local TV authentication so you of its 3 priority of annotation so you I am but can automatically look into ways of
those password passwordless as also the case in at at in that where I work we have categorized SSH them and I should say that chopsticks doesn't really have doesn't deal with the interactive that serve as a state so this also presupposes that you have to figure your
SSH infrastructure to be passwordless which I highly recommend anyway
we they get sense of who um I read chopsticks in the last year I came up the idea because I was grappling with principle
and school and and school is the um is going to afford is designed for the um and I was not using it for that so so I started to think of ways that I can do something better than like to wrap fabric which I used before and and that's why don't go back to the
to using fabric so this is this is a fabric scripts that I think is probably a couple of imports that needed to try this and this is a I of a recipe for fabric that I pulled straight out the that ball written so the and you would run this with a command fad upload so what you're actually writing here
in most of the operations is bashed so you were almost all of have also just using mostly run a little bit of sink and put and bashes point something so it's it's if you want to do anything more complicated than that you get placards territory of dropping a script over and a liking
it and um shipping results back in parsing and results we don't see that in in chopsticks um so the
compared to fabricate the chopsticks has a lot more flexibility it doesn't have the strength of execution model does not the found this a restriction you writing just whatever Python scripts you want to write um also fabric is that it does have parallelism there was much more of an afterthought so it works by the creating multiprocessing pool and running the operations on them um and there is no persistent state state on the other side so you and your her but if you do want to rerun things then I you'll boat you know run multiple commands then you having to restart the process every time and is also weighted to SSH
so fabric users parameter and parameter is a Python SSH library and and that means that it can only work the stages it is it is so that all the SSH where's chopsticks and is is pretty much agnostic about stages is using this same binaries but um as will see it's just sort of wrapping those and wrapped docket in exactly the same way and school um well this is the python
um this is so this is your main interface to transport some and you know people grow up like that was was written in Python but actually it is
a programming language written in animal some and Michael was literally like you can step through
the sequence of operations and I will substitute into the so you can get variables anxiously variables to the sine values variables and some shipment to the uh the next operation so
that it's it's like of much worse bash them the the Gospel modulate API we actually get to wanting to write someplace encouraged to extend Ansel some this is a little bit more
widely um the by I've wrestles with trying to write the things to the below API an echo of works but we came to the conclusion that an SVM is all about the Emil moment is not about Python alignment Python program so you um like presents those chopsticks gives you the ability to write normal testable Python encodes you could test a function lately and you
can have quite a high confidence that would is run remotely were remote system you could documented um by like about a project and merit where I 1st and then a class that I use both locally and remotely um so different parts of the of the operation um and uh it's uh as you can see then that made a place that is is not opinionated used in
these for writing things for monitoring to um harness the deployment to some so there are a range of things not as well as some of the orchestration Masaryk reconsideration management kind of operations um so they're really very few restrictions as to uh to how you can use chopsticks where whereas with a financeable you can only write 1 script for example chopsticks you can you have your entire code base available to impose on the right side so here works them the free tricks um and so do the 1st
trick is to 1st we have to get a Python process of running on remote site but we have to exchange of so is its victory
LPC so we need to make a request to it we need to get sponsors back um and then we need to have been able to import codes so can we get a Python process running that we can give a minus the option and pass like some a little script in the command
line and actually because we don't sends 10 kilobytes in in the command line so it's not come on line just reads a script um and then
once you got a larger script where where we can proceed but so but there was a focal bubble there's actually makes my nostrils and say like most the nationals and
toxic scratch out 6 this 1 was like that about you render those some a putting plastic plastic and rubber on a plate and you blow over a right and so
we inflate a process on the other side entirely with codes coming from the orchestration taste and the the command line in up on the 1st example is the kind of command line where and we we we feed it like a set of online version of the bootstrap script and then we pipe in the full script come and
that's a prefixed by SSH so that's that's just running that script were made haste and then because we got students stood out I can feel the re the requests from and receive responses Their those points so the
1st thing that will happen at the point is that since the reach of script and then that bootstrap script changes that played into a message passing protocol is the loops the students to them out of the way I'm and then we go bi-directional communication to the right place so that all the remote process which could be on the same text if some and that's the kind
of message protocol uh it's just like a standard binary protocol with all the trees
size data them and a message
type and a request ID for convenience to to the dispatcher to the right thing the right side um within that they're a serialization particles so the data of when
you operating a file that data is points the points of the file and the operation is provides the file and when you calling a function the the data will include the pinnacle of the the the code because and and then the last step is
importing code back from host and so for
this the RPC goes the other way so we make a cold thermal house and the remote host says are in order to empirical this function for example or in press the function contains imports I need to import this module and so do the other is an important hook which is a is a
capability in Python to customize the way impose and and it will go back to the orchestration Hasan say have you got this file I'm and the orchestration host will send this so importing on the right side just goes through the simple hook and all imports of pure Python stuff that will
just be delegated to the
to the right highest apart from hopefully the standard library will come from your own deployment of Python on the on the the remote site
said but having made with that work and I sort of moved on to doing so some of the uh what I'd have unlocked lots of
ideas to myself in how to extend this and 1 of the ideas I had was to enable this kind of configuration site of experience at at where I work I suggest that you can run maybe 20 SSH processes in comfortable comfortably
in parallel on the on the back of the account boxes so we have many wives um it is already possible and chopsticks to import chopsticks and make uh additional connection so that you can call a function that is defined to use chopsticks and you can say but 1 of the things that I'm hoping to write is
a full proxy for chopsticks that is entirely bootstrapped dynamically say the by the project that I'm trying to work moment where I need to connect to 750 hosts some so I should just be able to
connects to say 20 that that like pick 20 random or define another strategy for connecting to if for selecting trade case a connected them with chopsticks and all of those hosts import chopsticks and connects to the 1 in each of them connected 20 more Highsmith giving me 400 hosts a maybe like 3 numbers that will get deeper insights spanning tree of a connecting to host is this potentially achievable with obviously latency trader trade and so on on and I have run out of I but so this SSH
was was the the reason why invented this I wanted to do something that want to look like fabric manciple and Dr. I just suddenly realize what Dr. gives me access to students that have as well to Python process um and then city
city gives me students so that as long as there's no 2 possible prompts um so what else do I think parameters of about things that can give me a Python process so that is the sum of the context that some and speed my code to so here the links that I've got so extensive documentation on redox and the
project is on PI PI and get help the start of the Gulf stickers so if you would like a but there are also some parameters from the OK like those of the 2nd time and I will be spent this that we can't with the what would the
unit is what's
few reversal should be thank you for you told to so I was just wondering if you unless you have an interest for truth the diversity of servers you're using the SSH journaling yeah I it's it's that he if you have different versions of Python running on the different machines and you may have something not working from some of them do you manage when you return them you gets the you know at the single-molecule managed to analyze what's going on different servers and deeper would you knew that something has failed or that it I I didn't say error handling of Part I can find the candidate near my explain is what's and
in a way in the in the support that resolution and of who so to try to answer the without the project back again there also arose yet the um so when you are using a single tunnel it will raise exceptions when using a group of the results object you gets um a that contains the successes and failures in the failures of the container stack-trace um and the there is no access to the actual the frame objects and it was the site as the frame logics media stack-trace and if you are recursively telling you get a stack trace is collected so far this spectra schools the set price or something like that and so you can you can pick a part what happens I'm actually at the responsibility for error handling is mostly on the and chopsticks is not opinionated about how you might want to the surface these errors and so where the end school model for error handling is that when you get to narrow host you stop processing that hosts the and like the rest of the playbook will run on it and you might want to try and recover again there there's more questions coming from the latest thank you very much that it's also a 2 very short distance at was this is the way possible environment variables to the environment to which I'm calling and with his work if the line machine is the windows with working as going so you cannot possum armor variables that you could call a function that modifies the West and Brown of windows uh there is basically no support for Windows however the remote sites has been written to not use the kind of uh not use the kind a sink stuff but that is on the orchestration text so it would be possible to create a uh chopsticks agents for Windows um that type of thing if you configured it's some of your SSH if the configure Windows states of and Python correctly you might there would get it to work right now but but the thing that would be a an opportunity for future work uh um but I don't have anything
windows so I precedent tend to do that i if anybody else was contradicts then you can join in the sprints begin the height of the difference also library it's better than I
ever met in my peers to move used it therefore tested emissions it will buy stuff still exact and the ones that was kind of difficult is that obviously you can most colleges through because the remote I can never get the that is is it needs or perhaps something could be done on like a snapshot of the variables variables so the quotient means wouldn't think of that is that a good idea or what um so you might think it on uh the actually the way that the sterilization of those of Jupiter cells works is customer so that takes a snapshot of the labels and the the predicate as still they quite difficult due for closure um what can do is as a partials applicable after Python through 3 point something and say that it was a it was that was what about whether by some tools departure was not global it is now applicable and so that there is that I'm also you can integrate a class you could have a class that is picked picked labeled instances would be pixel that bound method is that the global so there are there always coupling of data to you you Cobles as well as a passing the of parameters we propose that we call tunnel local critical the there are 2 kind of sure they're related questions are not 1st and if I understood you correctly sometimes functions are shipped over the wire in source code form and sometimes Bishop over in some other form of water when you choose which form and always in the form of a case to so in if you if your script is running as under on the main is actually it's itself and on the main that the callable is defined that said you of cobalt got under under that under under module equals and run the main because those are not little wonderful OK so then you you do you do if you can you pick on the function and that's we should be aware the of so my the question is it seems like you work you have something very the about how you could run multiple Python versions depending on on what you want it necessarily on the on remote hosts and uterine were crossing the streams between 2 and 3 yes so so there are differences varied on differences about pickle between pattern to about 3 on did you just but not the attention to that or are you very intelligently converting ends up between the pickle formants on and what about like like the differences and things like that on it's so the invitation now that 1 of the 1st things it does is asks what's the holistic elections supported on the right side and it will use that call the collection so you and and actually to avoid some some problems we needed in a group of the uh when you connect a group of it which is the lowest common common denominator the collection for the entire group as mentioned something thank in Morrison's coming yet on the best in question is a variant of the to produce questions alone but so yeah if you consider loosening bicycle instead of the Inland Rules of Co so what we will have the same form of between 5 and 1 . 3 that maybe you know from my by to the a trial that I think there are too many problems in trying to use by a compiled the 1 version of Python with version Buffalo version of Python and I should say that even when your you're not on In the expected use cases this might make when you were a group of how the and expect to use this is this tragedy was once I the year using the
system Python interpreter so you are used in Python to or used in by the 3 the lecture tries to match the pattern to Python 3
to whichever system Python interpreter and you have available so that it is very likely that the minor versions weight match them and so do the traitors the
past by carriers is doomed to failure they think you know 1 last question this year and I what you also consider implementing this sounded dumb severance that the SSH server in order to for example of an Indian edition Windows yet I well that's that's why that I've already of the question but I think it's it's you follow them I regret it but it is it something like of about and the kind I have is so compatible with the same that's the that's tells the exception handling question get a remote exception such uh that has the
same as these parts of pods that's Kate imported from the orchestration this but if the user wants it
Gewichtete Summe
Gewichtete Summe
Gerichteter Graph
Streaming <Kommunikationstechnik>
Reverse Engineering
Skript <Programm>
Partielle Differentiation
Gleitendes Mittel
Kontextbezogenes System
Prädikat <Logik>
Rechter Winkel
Lesen <Datenverarbeitung>
Ordnung <Mathematik>
Algebraisch abgeschlossener Körper
Folge <Mathematik>
Klasse <Mathematik>
Mathematische Logik
Virtuelle Maschine
Spannweite <Stochastik>
Endogene Variable
Virtuelle Realität
Skript <Programm>
Ganze Funktion
Protokoll <Datenverarbeitungssystem>
Binder <Informatik>
Elektronische Publikation
Dämon <Informatik>
Konfiguration <Informatik>
Prozess <Physik>
Natürliche Zahl
Kartesische Koordinaten
Technische Optik
Einheit <Mathematik>
Prozess <Informatik>
Hook <Programmierung>
Schnitt <Graphentheorie>
Nichtlinearer Operator
Lineares Funktional
Prozess <Informatik>
Freier Ladungsträger
Funktion <Mathematik>
Strategisches Spiel
Projektive Ebene
Proxy Server
Web Site
Gewicht <Mathematik>
Zellularer Automat
Kombinatorische Gruppentheorie
Syntaktische Analyse
PERM <Computer>
Speicher <Informatik>
Physikalischer Effekt
Einfach zusammenhängender Raum
Elektronische Publikation
Physikalisches System
Migration <Informatik>
Objekt <Kategorie>
Serielle Schnittstelle


Formale Metadaten

Titel Scripting across hosts with Chopsticks
Serientitel EuroPython 2017
Autor Pope, Daniel
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/33694
Herausgeber EuroPython
Erscheinungsjahr 2017
Sprache Englisch

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Scripting across hosts with Chopsticks [EuroPython 2017 - Talk - 2017-07-13 - PythonAnywhere Room] [Rimini, Italy] Chopsticks lets your import and run Python code on remote Unix hosts over SSH. It works with no code deployment and no preinstalled software other than Python and SSH. It has built-in support for running code on many hosts in parallel. Unlike Fabric or Ansible, Chopsticks not opinionated about the structure of the code you run, allowing it to be used for orchestration, auditing, diagnostics, monitoring probes, and more Also unlike these, Chopsticks is not wedded to SSH, so it can transparently work on Docker containers over pipes, local subprocesses - and in future, sudo? Daniel will demonstrate Chopsticks in action, walk through the API, and explain the three clever tricks that Chopsticks is built on

Ähnliche Filme