Add to Watchlist

Embedding Python: Charming the Snake with C++


Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Embedding Python: Charming the Snake with C++
Title of Series EuroPython 2014
Part Number 91
Number of Parts 120
Author König, Michael
License CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
DOI 10.5446/20008
Publisher EuroPython
Release Date 2014
Language English
Production Place Berlin

Content Metadata

Subject Area Computer Science
Abstract Michael König - Embedding Python: Charming the Snake with C++ At the example of our in-house distributed scheduling system, we discuss the challenges of embedding the Python interpreter in a C++ program. Besides the actual integration of the interpreter, efficient data exchange between both languages is discussed. In addition, this presentation demonstrates how higher-level abstractions further diminish the language barrier. ----- Python with its huge standard library and sophisticated packages developed by its thriving community has become an incredibly useful tool for data scientists. At Blue Yonder, we value Python for the ease with which we can access and combine machine learning algorithms to build accurate prediction models. To get the most business value out of the use of Python, we strive to rid our model developers from all burdens outside their core expertise, i.e., developing statistical models. To leverage our existing infrastructure, essentially a distributed scheduling system written in C++, we decided to embed a Python interpreter in our application. The goal was to let developers use the language best suited for their problem, and to let them incorporate code created by others even if it is not written in the same language. In this presentation, I will talk about a few obstacles which we had to overcome in integrating the (C)Python interpreter in our C++ program, e.g., clean resource management, error handling, and broken features in the interpreter's API. I will show how we employed features from the [Boost Python C++ library] not only for simple data exchange, but also for more powerful concepts such as data sources. Finally, I will demonstrate how C++ objects can be used to seamlessly interact with Python, for example to use Python's logging package as usual while the actual logging is handled by our C++ application. With this combination of both worlds, we achieved a desirable mix of virtues: safe, reliable operations; good run-time performance; fast development; and highly expressive, unit testable core domain logic.
Keywords EuroPython Conference
EP 2014
EuroPython 2014

Related Material

OK hello everyone and today I will tell you a little about Python from a successful programmers perspective and and white school even for me and I am before I go right into it I just want to show you what I will talk about them 1st of all I will give you the short motivations quite high school inference users and I will tell you how to embed just the Python interpreter I will tell you how to interact with Python
code from 6 plus another giving you hints on what you can do to improve the the user experience so to say to include batteries and then I will finally come to the end of my talk of the short summary OK
so far beyond this data and company we do predictions for for clients and to
automated decision making and for this we have developed an in-house fiction platform and it accords distributed task scheduling system which also alternative view of things for example access to data sources such as databases and key-value stores and we've designed for reliable 24 7 operations and it's written in C + + and while platform simplifies and reduces many chores for model developers it's actually not the real helpful model development because C + + is a statically-typed typed language and iteration cycles tend to be pretty slow and library support is
not that good as somewhere so will look beyond C + + and basically immediately found python and the
python is a thriving community with great libraries and toolkits such a psychic learn which basically gives us machine learning algorithms for free and also provides a framework for putting our own reference in there
I can also be quite efficient by the use of optimized libraries such as number of
iron and and a call language of of Python provides us with a quick feedback loop you can just open your Python interpreter or in IPython notebook and just think about it and see what happens so this solution we opted for wants to embed Python and C + + to somehow combine 2 words we already have so safe reliable operations from us adjust application together with the possible development of Python analysis and we also get the added benefit the benefit that we are quick to production because there's no
need to redevelop the Python model in another language OK so how do we go
about integrating the Python interpreter and them this is probably the heartbeat of of the talk will suffer suppress postcode
at you and then there's a few
essentially the uh the Python reference implementation costs we need to make and what do we do and
privacy with basically created a context mentioned last and this is what I posited in + um the final clusters constructed which
corresponds to the and methods you you would have an important context manager and basically it's just called a supply initialized X function and to say the
signal handling this is useful to gain control over the Internet and of course the way government and convex mentally you also need to
have the structural or an and this exit method we just call it finalized method of C Python again some how would you use
the it's pretty simple you take an existing C + + program
for example here that the main function we just create an instance of this patent this would be similar to opening with broken Python and then you just do anything you want with the Python
interpreter so this is pretty neat but it doesn't come without habits because interpret initialization can be a problem that's well documented in the Python API so we tend to keep
just 1 interpreter life for the whole the lifetime of of process I think things become a
little more involved when we go to a thread environment and this is basically due to the global interpreter look see Python
implementation comes with and due to this we have to make some some adjustments and main right we have to make sure that the interpreter is actually aware
that our efforts might be using it and we have to release a global farther efforts to acquire and obviously when we use on other suggest threats we have to make sure that any time we execute Python code we acquire and release the
global look accordingly OK so what do we actually have to change this
is the python tuberculosis showed you before condensed form and basically we have to some operations to the construct 1st of all we have to call pi evil influence which makes the Python turbulent that where the other effects might be using it and then we have to save to to convert state for for later use this is done with the
idea of faith that function and when we say that they won't have to restore it at some point in this is done structure were recalled how
you will restore and to make sure everything is this is probably a so this is basically the only change you have to do for the main event and I more the threats we need another context mention score interpillar and in the constructor call applied to state and sure this would basically make sure that that's the that's the 1st successful threat to global interpolant is acquired and and he does everything itself us and
in the destruction of of this class we were released Tuesday it's just a matter of fact
I can and do that you
just go into your work effects which might be some function which is executed by the magistrate an instance of the global interval class
and this will hold chill slowness of lists and you can't do anything with the Python interpreter and I so
far I have explained how the actual embedding is done but I haven't shown yet how to interact with with the Python interpreter and for
interaction with founded whose hyphens suggest library most beneficial you can download for free on most of
which is essentially home of a collection of mature and commercially usable open-source software which yeah this is to go to place basically when you look for a C + + feature which is not
in the standard library again despite NASA lots lots of interesting feature which fields so
for example the threats lots of functions of C Python API and addresses for example to during the past this users of manual lifetime management of Ivan not object is automatically done for you but is also done for you is a type conversion between C + + and Python this was pretty convenient it also comes with some rudimentary exception handling support and there's a few convenient features to expose Python could suggest just go to Python so you can use the supplied from the sun to plus code Python just by using
important and the Senate Python syntax OK so have could
I evaluate some Python code with C + + and the 1st line basically introduces BP an abbreviation follicles Python much like you can do with the import statement then I define some some Python code which is
2 times 21 and to evaluate this we just evil function provided that was by Python and what we get is this
the Boost Python object as a result and to do anything in C + + with this whose Python objects left to can words to some C + + data
structure for example an integer and this is done with with the whose Python extract function
OK so this is the result would just be the integer 42 obviously a so we also have to add a little bit of error handling Canada and for this is typically common sense to to make created try catch block for try except that suits and Python and imposed by the with role and air already said exception
time it encounters an and is 1 of the weaknesses of course Python and the exception is pretty useless it doesn't contains spectra races all or the error message the type of data basically just indicates that an error has encountered and you could do something else voted for example at some improved
OK and this is why I suggested whenever you suppose Python features rapid again with some some decent and handling of your own I so that was the
basis of the interacting with the Python interpreter and now it's time to include the batteries to make
a Python uses have and I think this is probably the most important
slide of my talk a summary to just developers should always think of embracing 5 so whenever you have a who was a data structure and you spend months developing and um try to think how you can expose this and and just a
few deforestation always consider and 1 of differences make it a list
and the obvious the for this make it a dictionary because I mean by news laughter dictionaries and whenever you have more than just simple data you should check for existing standards and in the
Python community and for example when when exposed something which is range like
in user-centered iterated when you have a database connection and you want to pass it somehow to to Python for performance reasons so I want to take a look at the Python DB
API and this this effect for it thing and try to make it
look like like the Python object and this this leads to the to the to the to the general guideline that
Python code which you provided by users and you're executing embedded environment should never know 1 need to know that is embedded and so an example of this and I call this example looking out of the box the idea is what'd you will know
about Python so logging facility with the standard logging module this lecture in Python code here just type in what
logging and then something like blogging warning and and some aramid message and will be locked so what if we could not block this there are just too some some model users have to configure and Python what if we would automatically forward all log messages to C + + and also look into the same storage which we have configured into clusters so this is basically
removed it short to reduce that to configure the lock model both for C + + and and for Python and when when you know a bit about
logging modular this ideas to register custom loving handler which would do exactly this OK so How would look like we have some CPP Laura class on honesty pesticides which provides loss function and then we have to the lobbying Hendler
class on the Python size which provides basically the interface for and now is some of the bridge the gap between both worlds because if it's not trivial to implement C + + class which somehow implements a Python constant and we opt for the
solution which is quite often the to the best 1 and in an computer science we just add another layer of indirection so on Python side we use create another class called for to see he and this fall to see the
classes implements the logging Hendler interfaces so basically it just has to provide this amid function which
is about the only thing you really must provide following and and has this fall to CPP classes basically to receive records and pass it to some C + + reference which we have called for and then I would be officially that's encode a set with this the plus
+ parts of think about it where simple CPP Laura last it exposes nothing but very trivial block method which takes a message of a string and
just handed out to the console quite a stupid thing to do for logging them but you can replace it with anything that's that's more sophisticated now we have to expose this last Python sort of Python knows what is CP-logic and again we rely on the
Python to do exactly this we used to use Python class the template and they basically tell to to take this C + + last season log and expose 2 pi Python as a class of the names in the field as well this DP knowing basically indicates that the whenever you recalled initialize city the log
which should an exception so that's the Python users don't really initialize these this last themselves and finally the last line indicates that we want to expose to log of the function of security as the call methods the Python class feel OK so once this this code is from Python knows exactly what to make of Pacific Laura instance created in C + + and apartments we have
to define this this intermediate class this fall to CPP obviously needs to be derived from logging and led to satisfy all the necessary concepts and in the initialize so we take 1 argument as as the receiver and just stored in a a local variable and then we have to define the image functions just to to fill in the missing there which is missing from the standard logging and land and they might function would take some summary court understood the
past about about learning framework and we just extract the message from this record and pass it to to a receiver OK and so the
only thing that's left to do this we have to pass this instance of are exposed to Justice lasted Python class and we would have to do this in Python so 1st of all we create a C + + instance of a city class then we need to basically traverses through for the main model the dictionary of the main modules to finally obtain and afford to sit in the class and then all of
this will be a boost Python object called create and long the size and we can call this great and the function and by created by calling created Hendler and
passing into the see plus plus action or quite a lot of things happen who's Python we check for this is that the lower class if it's only registered for automatic type conversions because we have
done what I showed you 2 slides earlier this is the case so the translators class to Python objects with with and the signature of a specified form and this
object will be passed to the constructor initializes of forward to see the speed and so
it will know that their specific suggestions object living somewhere and that I can call the bracket operator called again and 1 surface send like can just use other methods to to register this Hendler was the from the logging framework and users can just use standard logging commands and everything the dialog will be forwarded to to suggest and this is fine because I didn't have to configure things twice I gave
may come to the end of my talk a short summary and what I wanted to take
home from in the last couple of minutes is a bit embedding Python it's not really that hard we have 2 basic API calls the API called would documented so they suggest there's no secret and no reason
not to fight and when you try that we have found that was Python helps actually it helps a lot
even though from time to time and maybe a little clumsy to use especially when it comes to 100 so you have some more just for yourself I think the key messages whenever you you are embedded Python interpreter make sure that the users of the embedded interpreter still allowed to write Python encode done force there you will see plus just data structures on them but
instead he had to to the well-established Python standards and and and conventions if you do this you use this would basically love you and
it is most important therefore is still maintain and a very important property namely unit of stability it against still run as the very same codon and your Python interpreter Python modules you can just use the the the Python unit testing wanted
to to test code and this is something you you basically have have to do because otherwise all the code is written in embedded in your interpreter will not be used in the future but so I if you have any
questions please ask now or just a standard of both and obviously we uninspiring thank you so hi and such that it is little
things like this and you you should be using Python protocol stack and the tension members have on and uh so I'm a little surprised that using the agency is was higher than for embedding of and addition of a various supply starts centric view was transfer somewhat of a contradiction to the army to the use of this is very example for example which is which your grocery Prof treated different pieces of coordinates can the context so that it to the standard model wages with Python embedding something very puzzling right that south on them that is so and we
decided to use is Python mainly because the British Library is heavily used in our approach um so there's there's lots of other modules that there um and it's been around for for
quite a while so and we were pretty sure is that it works and to be honest I didn't too much of a problem so far so I mean
inference from the perspective of python uses they don't really know that they're running in spite of C just the it's natural for them for me as a C + + development both Python fits naturally and language to program in all library to provide them with so does that I don't see this this this however problem for a C + + programming is suppressed you we can use that but it still fits by funding for for the Python's I don't see the need for C + + programming to use more Python that is required 1st 1 is a and glad to more these Python it's it's a
beautiful library I think deeply underappreciated in the by the community so I 2 questions the 1st person to phone and so on 1 thing I noticed you said you in your Python Python interpreter classes destructors calling pi finalized that last time I checked is actually not supposed to be done if you're using these Python because of synthetic the initialization issues might occur you guys not seeing any problems with my finalize words it's not an issue for you um it's it's not really an
issue for us because we only do it once in a while and seen issues when you use multiple times and and and you're right there are some some issues with set things it depends on on on really what to do but so far we base here from in fact was stopped Python English interpreter and now it's all tied up correctly I mean we don't see segmentation faults and when we get a program so OK that's that's all fine and I guess my 2nd
thoroughly questioning is more appointed there's a library called backward to C + + library is designed to use simplified embedding Python and C + + and exposes a lot of things like you know of whites are objects advisory objects in and the loading library for instance is exposed as he was wasn't in fact you can create supporting him a subclasses directly using response without any infrastructure in Python using this library and have you guys have you every record of it never used it at all I actually haven't heard of it
gets sounds interesting and I would definitely talk to you all of the fact that density information I can get together I give thanks a lot could explain it my you've chosen to
and that type in C + + rather than writing music this was called and make it available as a Python module the yeah I compared to
liberate the timing and I mentioned that basically we have this toss doing systems and that's called legacy projects in the US in development and stuff like this and they they've already written in C + + and that all C + + past and he thought about that it's it's probably easier to embed a Python interpreter some separate plug into our framework and actually was easy them it was like that and that the basic prototypes student like 2 or 3 years of and cost and you liberate it it also gives us additional benefits and we don't have to rewrite also + code we can use the existing some of the infrastructure which was already adapted to to do the model for executable to command line parameters consideration we can we can pass and it also allows us to to make C + + and Python task so I can consider the after-school optimized C + + code return something and then will ultimately become words to to the to the Python code in Python can then use some cool transformations and just return it to the next person job and you can you can change without even knowing that despite something it's an easier topic another reminding readers
this discussion this thing so some some people can kind of think that embedding is something known from the very different from from from from extending and and so we can have provide a model or independent and embed python it's so the the only real difference is who starts the hyper runtime and from that point on it's exactly the same thing what difference anymore only because they have interaction between justice and content and have hundreds of thousands of both original and so this is really just the normal course parish lies on the use of to have interpreted as if the person on the command line runs it where it was supposed response and from that point on regional difference in words exactly the same kind of code and behavioral since the something thanks a remarkable I just speak
about a little bit it's 9 so we said is true but there are some
really substantial differences between the 2 1 is your management which you brought up is something that I can that EXE does for you or Python and Linux and but then you have to actually take direct control of an intermediate situation and you don't have to take care of so I have to take
care of the do management as soon as you have a thread started by C + + right when you use the grating and Python so started in the Python interpreter and this is automatically taken care of you know you this is basically just something we had to do because we basically handle requests and separate its moment right right I agree I agree with you were saying the same
thing I think the other substantial differences in this this because this is a development issue is that he can't be attached to embedded Python very easily you have to insert things you can't start sequence + program under the control of PDB and this is actually a strong argument in some cases for not using embedding or in fact take your C + + programming making it into an portable module even if it's you know 500 megabytes is something you can do so you can actually get multistatic debugging more easily so there's no substantial difference i've seen in cases were embedding has been used OK so I'm just click on
that ad by the argument devalues the difference we have embedded interpreter can
general got the commandments of the stuff from trading however is not different at all because you can just run superstar threats from uh and um embedded in the present update and to start up because it was assumed that somewhere and American well 7 1 1 instance modules in Python and the source of the signal strength somewhat different than at the crossing the same thing the only point is that you have to manage the deal if you do that and that's that's the only the only difference game of killings
follows questions accusing you talk is soaring to think I'm PLO
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation


  510 ms - page object


AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)