Embedding Python: Charming the Snake with C++
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 91 | |
| Number of Parts | 119 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/20008 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Berlin |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
Interpreter (computing)InferenceProgrammer (hardware)Perspective (visual)BitCodeComputer animationLecture/Conference
00:43
Metropolitan area networkDisintegrationActive contour modelComputer fontComputing platformPredictionPhysical systemTask (computing)Scheduling (computing)Source codeOperations researchWritingData modelOnline helpReal numberCodePredictabilityClient (computing)Library (computing)Decision theoryEndliche ModelltheorieSoftware developerView (database)Online helpIterationPhysical systemComputing platformCycle (graph theory)Data storage deviceOperator (mathematics)Formal languageExterior algebraDatabaseScheduling (computing)Source codeReal numberKey (cryptography)Core dumpJSONXMLLecture/Conference
01:46
Metropolitan area networkUniform resource nameAlgorithmVirtual machineFreewareVirtual machineSoftware frameworkAlgorithmLecture/ConferenceJSONXMLUML
02:08
AlgorithmVirtual machineLoop (music)FeedbackFormal languageCore dumpOperations researchData modelMetropolitan area networkAutonomous System (Internet)FeedbackSystem callEmbedded systemMathematical analysisWordProduct (business)LaptopInterpreter (computing)Software developerFormal languageOperator (mathematics)Cartesian coordinate systemEndliche ModelltheorieCore dumpLecture/ConferenceJSONXML
02:50
Interpreter (computing)DisintegrationMetropolitan area networkStrutEndliche ModelltheorieFormal languageInterpreter (computing)CodeBitImplementationLecture/ConferenceUMLJSONXML
03:14
Interpreter (computing)StrutContext awarenessMetropolitan area networkData managementContext awarenessInformation privacyGene clusterCorrespondence (mathematics)Functional programmingLecture/ConferenceComputer animation
03:36
StrutInterpreter (computing)System callGraphics tabletRadio-frequency identificationSurjective functionPattern languageUniform resource nameTotal S.A.Game controllerConvex setInternetworkingContext awarenessInterpreter (computing)Computer programmingFerry CorstenLecture/ConferenceComputer animation
04:01
Metropolitan area networkPattern languageInterpreter (computing)Instance (computer science)Functional programmingInterpreter (computing)Block (periodic table)Lecture/ConferenceComputer animation
04:27
Integrated development environmentInterpreter (computing)Process (computing)Video gameInterpreter (computing)Thread (computing)Integrated development environmentImplementationRight angleLecture/ConferenceJSONXMLUML
04:54
Metropolitan area networkIntegrated development environmentInterpreter (computing)Thread (computing)Context awarenessCodeMultiplication signInterpreter (computing)Thread (computing)JSONXMLUMLLecture/Conference
05:16
Interpreter (computing)StrutContext awarenessRadio-frequency identificationState of matterThread (computing)Social classState of matterInterpreter (computing)Thread (computing)Functional programmingOperator (mathematics)Context awarenessOcean currentConstructor (object-oriented programming)Form (programming)Sound effectMultilaterationSystem callTurbulenceXML
05:38
Interpreter (computing)Context awarenessThread (computing)State of matterRadio-frequency identificationMetropolitan area networkInterior (topology)Point (geometry)Data structureFunctional programmingAnnihilator (ring theory)Thread (computing)Event horizonConstructor (object-oriented programming)InterpolationSystem callMathematicsContext awarenessInterpreter (computing)Modal logicState of matterLecture/ConferenceXML
06:16
State of matterStatisticsContext awarenessInterpreter (computing)StrutThread (computing)Pattern languageOvalSocial classInstance (computer science)Functional programmingInterpreter (computing)Sound effectThread (computing)Electronic mailing listLecture/ConferenceXML
06:46
Active contour modelInterpreter (computing)Library (computing)UsabilityOpen sourceInterpreter (computing)Einbettung <Mathematik>Library (computing)Interactive televisionCapability Maturity ModelUsabilityOpen sourceSoftwareUMLLecture/ConferenceJSONXML
07:17
Library (computing)Open sourceUsabilityWrapper (data mining)Object-oriented programmingData managementVideo gameData conversionData typeException handlingCodeField (computer science)Library (computing)CodeReading (process)Exception handlingFunctional programmingData conversionObject-oriented programmingData managementType theorySpecial unitary groupAddress spaceLecture/ConferenceJSONXML
08:05
CodeState diagramComputer clusterMetropolitan area networkCodeLine (geometry)Statement (computer science)Multiplication signLecture/ConferenceXML
08:27
CodeRing (mathematics)State diagramMultiplication signFunctional programmingObject-oriented programmingResultantWordIntegerData structureLecture/ConferenceXML
08:53
CodeState diagramMetropolitan area networkIntegerBlock (periodic table)ResultantBitSuite (music)Exception handlingError messageXML
09:16
Error messageException handlingVulnerability (computing)Multiplication signType theoryLecture/Conference
09:38
Mountain passState diagramError messageCodeMetropolitan area network3 (number)Exception handlingInterpreter (computing)Basis <Mathematik>Multiplication signSlide ruleComputer animationLecture/Conference
10:02
Data structureData structureSlide ruleSoftware developerDefault (computer science)Different (Kate Ryan album)Electronic mailing listJSONXMLUMLLecture/Conference
10:24
Data structureElectronic mailing listMetropolitan area networkDefault (computer science)Data dictionaryIterationDatabaseInterface (computing)Connected spaceJSONXMLUMLLecture/Conference
10:57
Data structureCodeElectronic mailing listSound effectReal numberObject-oriented programmingPauli exclusion principleCuboidIntegrated development environmentCodeEinbettung <Mathematik>Lecture/ConferenceXML
11:28
CuboidMessage passingCodeType theoryModule (mathematics)Message passingError messageData storage deviceEndliche ModelltheorieGene clusterLecture/Conference
12:05
Message passingCuboidSocial classEndliche ModelltheorieBitFunctional programmingSocial classInsertion lossLecture/ConferenceComputer animation
12:36
Execution unitSocial classCuboidMessage passingMetropolitan area networkMobile appRow (database)Logical constantSocial classBridging (networking)Interface (computing)Computer scienceInteractive televisionLecture/ConferenceXML
13:12
Social classCuboidLogarithmRow (database)Message passingOvalState diagramMetropolitan area networkFunctional programmingInterface (computing)Social classTask (computing)String (computer science)Message passingCodeMereologyVideo game consoleBlock (periodic table)Fiber bundleLecture/ConferenceComputer animation
13:53
System callStrutMessage passingOvalState diagramLogarithmCAN busMetropolitan area networkQuicksortVideo game consoleSocial classLoginField (computer science)Template (C++)Lecture/ConferenceXML
14:38
Line (geometry)Social classSystem callException handlingFunctional programmingCodeInstance (computer science)Information securityLecture/Conference
15:12
CAN busSocial classInterior (topology)Metropolitan area networkRow (database)Variable (mathematics)Parameter (computer programming)Row (database)Functional programmingMedical imagingSocial classXML
15:45
Instance (computer science)Social classLogarithmModule (mathematics)Software frameworkMessage passingEndliche ModelltheorieSocial classData dictionaryInstance (computer science)Module (mathematics)Object-oriented programmingResultantLecture/ConferenceXML
16:28
Metropolitan area networkSocial classInstance (computer science)LogarithmModule (mathematics)Maxima and minimaObject-oriented programmingFunctional programmingGroup actionSocial classData conversionType theoryLecture/ConferenceComputer animation
16:52
CASE <Informatik>Electronic signatureForm (programming)Object-oriented programmingSocial classTranslation (relic)Slide ruleConstructor (object-oriented programming)Poisson-KlammerSoftware frameworkOperator (mathematics)SurfaceLecture/Conference
17:42
EmulationBitEinbettung <Mathematik>System callOnline helpComputer animationJSONXMLUMLLecture/Conference
18:11
WritingCodeForcing (mathematics)Interpreter (computing)Multiplication signData structureMessage passingWritingException handlingCodeJSONXML
18:36
WritingCodeExecution unitUnit testingStability theoryInterpreter (computing)Execution unitCategory of beingCodeModule (mathematics)Lecture/ConferenceJSONXML
19:01
Interpreter (computing)CodeLecture/ConferenceComputer animation
19:28
InfinityCommunications protocolComplex (psychology)CodeDifferent (Kate Ryan album)BitEinbettung <Mathematik>Heat transferStandard ModelView (database)Stack (abstract data type)AdditionRight angle
20:28
CuboidSocial classLogarithmRow (database)Module (mathematics)CodeLibrary (computing)Lecture/ConferenceComputer animation
20:51
Software developerLibrary (computing)Perspective (visual)Formal languageProgramming paradigmProgrammer (hardware)Computer programmingInferenceNatural numberLecture/Conference
21:34
Library (computing)Multiplication signSocial classInterpreter (computing)Fluid staticsWordLecture/Conference
22:06
Thread (computing)Interpreter (computing)Fluid staticsComputer programmingLecture/Conference
22:35
Dependent and independent variablesRow (database)Instance (computer science)Einbettung <Mathematik>Object-oriented programmingLibrary (computing)Pointer (computer programming)
23:05
InformationPopulation densityEmbedded systemBitType theoryMusical ensembleCodeLecture/Conference
23:27
System programmingSoftware frameworkTask (computing)WordProcess (computing)Software developerInterpreter (computing)CodeMultiplication signProjective planeTransformation (genetics)Student's t-testPrototypeConfiguration spacePlug-in (computing)File formatMixed realityPhysical systemScheduling (computing)Lecture/Conference
24:52
Different (Kate Ryan album)Embedded systemPoint (geometry)WordDialectNormal (geometry)Content (media)Endliche ModelltheorieCodeInteractive televisionEinbettung <Mathematik>Independence (probability theory)Module (mathematics)Interpreter (computing)Direction (geometry)Run time (program lifecycle phase)
25:46
BitDifferent (Kate Ryan album)Data managementEinbettung <Mathematik>Game controllerThread (computing)Flow separationInterpreter (computing)Moment (mathematics)Lecture/Conference
26:31
Computer programmingSoftware developerSequenceParameter (computer programming)Different (Kate Ryan album)Einbettung <Mathematik>CASE <Informatik>Game controllerModule (mathematics)
27:03
Parameter (computer programming)Interpreter (computing)Different (Kate Ryan album)Module (mathematics)Thread (computing)Point (geometry)Game theoryInstance (computer science)Source codeLecture/Conference
27:43
GoogolLecture/Conference
Transcript: English(auto-generated)
00:15
Okay, hello everyone. Today I will tell you a little bit about Python from a C++ programmer's perspective
00:23
and why it's cool, even for me. And before I go right into it, I just want to show you what I will talk about. First of all, I will give you a short motivation why Python is cool, even for C++ users. Then I will tell you how to embed a CPython interpreter.
00:41
I will tell you how to interact with Python code from C++, and I will give you hints on what you can do to improve the user experience, so to say, to include batteries. And then I will finally come to the end of my talk with a short summary. Okay, so Blue Yonder is a data-driven company.
01:03
We do predictions for clients and do automated decision-making, and for this we have developed an in-house prediction platform, and at the core it's a distributed task scheduling system, which also handles a few other things, for example, access to data sources such as databases
01:21
and key value stores. We've designed it for reliable 24-7 operations, and it's written in C++. And while our platform simplifies and reduces many chores for model developers, it's actually not a real help for model development, because C++ is a statically-typed language, and iteration
01:45
cycles tend to be pretty slow, and library support is not that good as somewhere else. So we looked beyond C++ and basically immediately found Python, and Python is a thriving community with great libraries and toolkits such as scikit-learn, which basically gives us machine
02:04
learning algorithms for free, and also provides a framework for putting our own algorithms in there. Python can also be quite efficient by the use of optimized libraries such as NumPy and Pandas, and the core language of Python provides us with a quick feedback loop.
02:22
You can just open your Python interpreter or an IPython notebook and just tag about and see what happens. So the solution we opted for was to embed Python in C++ to somehow combine the words we already have, so the safe, reliable operations from our C++ application, together
02:41
with the fast model development Python allows us. And we also get the added benefit that we are quick to production, because there's no need to redevelop the Python model in another language. Okay, so how do we go about integrating the Python interpreter?
03:01
And this is probably the hard bit of the talk where I start throwing C++ code at you. And there's a few essential C Python reference implementation calls we need to make, and what you would do in Python is you would basically create a context manager class, and this is what I also did in C++.
03:21
In C++, we define a C++ constructor, which corresponds to the enter method you would have in a Python context manager, and you basically just call this Py initialize X function and disable signal handling. This is useful to gain control over the interpreter.
03:43
And of course, where you have an enter in a context manager, you also need to have a destructor or an exit method, and this exit method we just call the Py finalize method of C Python. Okay, so how would you use this? It's pretty simple.
04:00
You take an existing C++ program, for example, here the main function, you just create an instance of this Python interpreter class. This would be similar to opening a with block in Python, and then you just do anything you want with the Python interpreter. So this is pretty neat, but it doesn't come without caveats, because interpreter reinitialization
04:23
can be a problem that's well documented in the C Python API, so we tend to keep just one interpreter alive for the whole lifetime of our process. Okay, things become a little more involved when we go to a threaded environment, and this is basically due to the global interpreter lock, the C Python implementation comes with,
04:46
and due to this, we have to make some adjustments, and in the main thread, we have to make sure that the interpreter is actually aware that other threads might be using it, and we have to release the global interpreter lock for other threads to acquire.
05:02
And obviously, when we spawn other C++ threads, we have to make sure that any time we execute Python code, we acquire and release the global interpreter lock accordingly. Okay, so what do we actually have to change? This is the Python interpreter class I showed you before in a little condensed form, and
05:21
basically we have to add some operations to the constructor. First of all, we have to call PyEvalInThreads, which makes the Python interpreter aware that other threads might be using it. And then we have to save the current thread state for later use. This is done with the PyEvalSaveThread function.
05:41
And when we save a thread state, we also have to restore it at some point, and this is done in a destructor where we call PyEvalRestoreThread to make sure everything is properly matched. Okay, so this is basically the only change we have to do for the main thread, and for the other threads, we need another context manager, let's call it global interpreter
06:03
lock, and in the constructor, we call PyJilStateEnshore. This will basically make sure that there's a thread state for the C++ thread, the global interpreter lock is acquired, and it does everything that's necessary for us. And in the destructor of this class, we will release the JilState just to match
06:22
the bracket, basically. Okay, and to use it, you would just go into your worker thread, which might be some function which is executed by a thread, and just create an instance of the global interpreter lock class, and this will hold the Jil as long as it lives, and you can do anything
06:41
with the Python interpreter in this thread. Okay, so far, I've explained how the actual embedding is done, but I haven't shown you yet how to interact with the Python interpreter. And for interaction, we have found the Boost Python C++ library most beneficial.
07:03
You can download it for free on Boost.org, which is essentially home of a collection of mature and commercially usable open source software, which is the go-to place, basically, when you look for a C++ feature which is not in the standard library.
07:22
Okay, and Boost Python has lots of interesting features which we use. For example, it writes lots of functions of the CPython API, and it writes it, for example, to read C++ users of manual lifetime management of Python object. This is automatically done for you.
07:42
What it's also done for you is a type conversion between C++ and Python. This is pretty convenient. It also comes with some rudimentary exception handling support, and there's a few convenient features to expose C++ code to Python, so you can use C++ code in Python
08:05
just by using import and the standard Python syntax. Okay, so how could I evaluate some Python code with C++? The first line basically introduces BP as an abbreviation for Boost Python, much like
08:22
you can do with the import statement. Then I define some Python code, which is two times 21, and to evaluate this, we use the eval function provided by Boost Python, and what we get is a Boost Python object as a result. And to do anything in C++ with this Boost Python objects, we have to convert it to
08:45
some C++ data structure, for example, an integer, and this is done with the Boost Python extract function. Okay, so CPP result would just be the integer 42, obviously. Okay, so we also have to add a little bit of error handling, and for this, it's
09:06
typically common sense to create a try-catch block, or try-accept, as you would say in Python, and Boost Python will throw an error-already-set exception any time it encounters an error. And here's one of the weaknesses of Boost Python, the exception is pretty useless,
09:23
it doesn't contain stack traces, or the error message, or the type of the error, it basically just indicates that an error has been encountered, and you could do something else about it, for example, add some improved error handling. Okay, and this is why I suggest that whenever you use Boost Python features,
09:43
wrap it again with some decent error handling of your own. Okay, so, that was the basics of interacting with the Python interpreter. Now it's time to include the batteries to make Python users happy. And I think this is probably the most important slide of my talk, besides the summary.
10:06
C++ developers should always think of embracing Python. So whenever you have a cool C++ data structure and you spend months developing it, try to think how you can expose this to Python. And there's a few defaults you should always consider,
10:22
and one of the defaults is make it a list, and the other obvious default is make it a dictionary, because, I mean, Python users love the dictionaries. And whenever you have more than just simple data, you should check for existing standards in the Python community.
10:42
For example, when you expose something which is range-like, then use the standard iterator interface. When you have a database connection and you want to pass it somehow to Python for performance reasons or whatsoever, take a look at the Python DB API, there's a pep for it, I think, and try to make it look like a real Python object.
11:06
And this leads basically to the general guideline that Python code, which you are provided by users and you execute in your embedded environment, should never know or need to know that it's embedded.
11:21
And I will show you an example of this. And I call this example logging out of a box. The idea is most of you will know about Python's logging facility with a standard logging module. There's actually some Python code here. You just type import logging, and then something like logging dot warning,
11:43
and some error message, and it will be locked. So what if we could not lock this error just to some logger, users have to configure in Python? What if we would automatically forward all our messages to C++ and also lock it to the same storage which we have configured in C++?
12:04
So this would basically remove the chore to configure the lock module both for C++ and for Python. And when you know a bit about the logging module, the obvious idea is to register a custom logging handler which would do exactly this.
12:26
Okay, so how would it look like? We have some CPP logger class on the C++ side which provides log function, and then we have the logging handler class on the Python side which provides basically the interface for all logging.
12:43
And now we somehow have to bridge the gap between both worlds because it's not trivial to implement the C++ class which somehow implements the Python concept. And we are up for the solution which is quite often the best one in computer science.
13:04
We just add another layer of interaction. So on the Python side we create another class, let's call it forward to CPP. And this forward to CPP class implements the logging handler interface. So basically it just has to provide this emit function
13:21
which is about the only thing you really must provide for logging handler. And the task of this forward to CPP class is basically to receive records and pass it to some C++ reference which we have stored before. And I will briefly show you how this looks in code.
13:42
I start with the C++ part. So think about we have a simple CPP logger class. It exposes nothing but a very trivial log method which takes a message as a string and just prints it out to the console. So this is quite a stupid thing to do for logging.
14:01
But you can replace it with anything that's more sophisticated. And now we have to expose this class to Python so that Python knows what a CPP logger is. And again we rely on Boost Python to do exactly this. We use the Boost Python class template. And we basically tell it to take the C++ class CPP logger
14:25
and expose it to Python as a class of the name CPP logger as well. This BP basically indicates that whenever we call the initializer of the CPP logger it should throw an exception so that Python users don't really
14:45
initialize this class themselves. And finally the last line indicates that we want to expose the lock function of our CPP logger class as the call method of the Python class CPP logger.
15:03
Okay so once this code is run Python knows exactly what to make of a CPP logger instance created in C++. And on the Python end we have to define this intermediate class, this forward to CPP. Obviously it needs to be derived from logging handler to satisfy all the necessary concepts.
15:25
In the initializer we take one argument as a receiver and just store it in a local variable. And then we have to define the emit function just to fill in the missing gap which is missing from the standard logging handler.
15:42
And the emit function will take some record. This will be passed to you by the logging framework. And we just extract the message from this record and pass it to our receiver. Okay and so the only thing that's left to do is we have to pass this instance of our exposed C++ class to the Python class.
16:06
And we would have to do this in Python. So first of all we create a C++ instance of our CPP logger class. Then we need to basically traverse through the main module, the dictionary of the main module to finally obtain the forward to CPP class.
16:27
And the end result of this will be a Boost Python object called create handler on the CPP side. And we can call this create handler function. And by calling create handler and passing it to C++ action logger
16:42
quite a lot of things happen. Boost Python will check for the CPP logger class if it's already registered for automatic type conversion because we have done what I've shown you two slides earlier. This is the case. So to translate this class to Python object with the signature as I specified before.
17:04
And this object will be passed to the constructor or initializer of forward to CPP. And so it will know that there is a C++ object living somewhere and that I can call the bracket operator to call it.
17:21
Okay and once I have the handler I can just use other methods to register the handler with the logging framework. And users can just use the standard logging commands and everything they log will be forwarded to C++. And this is fine because you don't have to configure things twice.
17:43
Okay let me come to the end of my talk. It's a short summary. And what I want you to take home from the last couple of minutes is that embedding Python is not really that hard. We have the basic API calls. The API calls are well documented.
18:01
So basically there's no secret and no reason not to try it. When you try though we have found that Boost Python helps. Actually it helps a lot. Even though from time to time it may be a little clumsy to use especially when it comes to error handling. So you have to somehow add this for yourself.
18:22
And I think the key message is whenever you embed a Python interpreter make sure that your users of the embedded interpreter are still allowed to write Pythonic code. Don't force your C++ data structures on them but instead adhere to the well established Python standards and conventions.
18:42
If you do this your users will basically love you. And if you do this most importantly of all you still maintain a very important property. Namely unit testability. If you can still run the very same code in your Python interpreter or your IPython module you can just use the Python unit testing module to test your code.
19:04
And this is something you basically have to do because otherwise all the code that is written and embedded in your interpreter will not be used in the future. Okay so if you have any questions please ask them now
19:21
or just visit us down at our booth and obviously do your under-assiring. Thank you. Hi, so you said a couple of things like okay so you should be using Python protocols
19:42
and you should embrace Python and so I'm a little surprised that you're using so you chose to use Boost Python for embedding Python and that actually has a very C++ centric view so that sounds like a somewhat of a contradiction to me.
20:04
So you showed this logging example for example which goes through kind of three different pieces of code and kind of complex. So could you explain a bit why you choose Boost Python
20:20
instead of embedding something very Pythonic like Cython? Okay so we decided to use Boost Python mainly because the Boost library is heavily used in our code so there's lots of other modules there and it's been around for quite a while
20:42
so we were pretty sure that it works and to be honest it didn't prove too much of a problem so far. So I mean from the perspective of our Python users they don't really know that they're running inside of C++ it just feels natural for them.
21:02
For me as a C++ developer Boost Python feels naturally a language to program in or a library to program with so I don't see this paradigm problem. A C++ programmer says C++ view, he can use that but it still feels Pythonic for the Python user.
21:23
I don't see the need for a C++ programmer to use more Python than is required. So first I just want to say I'm glad to see more Boost Python it's a beautiful library and I think deeply underappreciated in the Python community.
21:41
So I had two questions the first is, first answer the phone. One, I noticed you said in your Python interpreter class your instructor was calling pi finalize but that last time I checked is actually not supposed to be done if you're using Boost Python because there's some static de-initialization issues that might occur
22:01
are you guys not seeing any problems with pi finalize or is it just not an issue for you? It's not really an issue for us because we only do it once. I've seen issues when you do it multiple times and you're right there are some issues with static things it depends on really what you do but so far we basically have our main thread
22:23
the Python interpreter there and it's all tidied up correctly we don't see segmentation faults when we quit our programs that's all fine. Okay, and I guess my second, it's not really a question I guess more of a pointer there's a library called Akwork it's a C++ library
22:41
and it's designed to simplify embedding Python into C++ and it exposes a lot of things like bytes or objects and bytes array objects in it and the logging library for instance is exposed into C++ and in fact you can create logging handler subclasses directly in C++ without any infrastructure in Python
23:00
using this library and have you guys, have you ever even heard of it and have you used it at all? Actually I haven't heard of it yet it sounds very interesting and I will definitely talk to you afterwards I'll just track you down and talk about it later To leech all the information I can get Yeah, okay, thanks Okay, thanks a lot Could you explain a bit why you've chosen to embed Python in C++ rather than wrapping your C++ code
23:22
and make it available as a Python module? Yeah, I can try to elaborate a bit I mean, I mentioned that basically we have this really tasks scheduling system and that's called legacy projects even though they are still in development and stuff like this
23:41
they were already written in C++ and they had all those C++ tasks and we thought about that it's probably easier to embed the Python interpreter in some separate plugin to our framework and actually it was easy because it was like done
24:00
the basic prototype stood in like two or three hours and of course then you elaborate on it and it also gives us additional benefits I mean, we don't have to rewrite all our C++ code we can use our existing other infrastructure which was already adapted to the format of our executable
24:20
to the command line parameters to the configuration we can pass to it and it also allows us to mix C++ and Python tasks so I can say, okay, here after school optimize C++ code it will return something and then it will automatically be converted to the Python code and Python can then do some cool transformations
24:41
and just return it to the next C++ job and you can chain it without even knowing that there's Python in there so it seemed easier. I have another remark regarding this question so some people kind of think that embedding is something very different from extending
25:04
and so you can either provide a module or you can embed Python so the only real difference is who starts the Python runtime and from that point on it's exactly the same thing no difference anymore because you have interaction between C++ and Python
25:23
and Python and C++ in both directions and so it's really just who calls pyinitialize who starts up the Python interpreter is it the person on the command line who runs it or is it the C++ code starting it and from that point on it's really no difference anymore it's exactly the same kind of code and behavior
25:43
and it's the same thing thanks for the remark can I just speak to that a little bit it's 90% of what you said is true but there are some really substantial differences between the two one is GIL management which he brought up is something that python.exe does for you or Python and Linux
26:01
but then you have to actually take direct control of in an embedding situation you only have to take care of oh sorry you only have to take care of the GIL management as soon as you have threats started by C++ when you use threading in Python so it started in the Python interpreter and this is all automatically taken care of for you
26:21
this is basically just something we had to do because we basically handle requests in separate threads in our main thread right now I agree with you we're saying the same thing and I think the other substantial difference and this is a development issue is that pdb can't be attached to embedded Python very easily you have to insert things
26:40
you can't start a C++ program under the control of pdb and this is actually a strong argument in some cases for not using embedding or in fact taking your C++ programming making it into an importable module even if it's you know 500 megabytes this is something you can do so that you can actually get multi-stack debugging more easily so that's another substantial difference I've seen in cases where embedding has been used
27:03
so just a quick remark I buy the argument that debugging is a bit different when you have an embedded interpreter and can't just you know go to the command line and stop it and do stuff threading however is not a difference at all because you can just run C++ threads
27:20
from an embedded Python interpreter you can just start up a C++ thread somewhere and the other way around as well so you have a C++ module in Python and that starts up a C++ thread so no difference there I think we're all saying the same thing the only point is that you have to manage the GIL if you do that and that's the only difference
27:45
okay thanks for all those questions okay thank you also for your talk that's very interesting and applause