Demystifying Python’s Internals
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 112 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 4.0 International: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben. | |
| Identifikatoren | 10.5446/60790 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
EuroPython 202225 / 112
5
10
14
17
19
23
25
29
31
32
34
44
47
51
53
54
57
61
69
70
82
83
88
93
94
97
101
105
106
00:00
Nichtlinearer OperatorGoogolMultiplikationsoperatorSoftwareentwicklerDean-ZahlPhysikalisches SystemVorlesung/Konferenz
00:31
QuellcodeNebenbedingungNichtlinearer OperatorToken-RingFormale GrammatikParserAbstraktionsebeneAbstrakter SyntaxbaumBefehlscodeCompilerLeistungsbewertungLoopReelle ZahlSoftwareentwicklerElektronischer ProgrammführerMereologieMultiplikationsoperatorDatenkompressionSoftwareentwicklerSchnelltasteVersionsverwaltungFormale GrammatikInformationParserNichtlinearer OperatorAbstrakter SyntaxbaumImplementierungCompilerGemeinsamer SpeicherSpieltheorieProdukt <Mathematik>ProgrammierungQuellcodeRechenschieberLoopLeistungsbewertungProgrammierspracheToken-RingDokumentenserverBitElektronischer ProgrammführerDivergente ReiheFunktionalNebenbedingungThumbnailFormale SpracheZahlenbereichParametersystemQuick-SortAbstraktionsebeneMultipliziererComputeranimation
04:00
QuellcodeNichtlinearer OperatorBinärdatenImplementierungMereologieAbstrakter SyntaxbaumStreaming <Kommunikationstechnik>Token-RingElektronische PublikationFormale GrammatikDatenkompressionFunktionalNichtlinearer OperatorEinfache GenauigkeitResultanteParametersystemGeradeInternetworkingDokumentenserverSchlussregelMapping <Computergraphik>SystemaufrufMereologieElektronische PublikationFormale SpracheOSS <Rechnernetz>BinärcodeToken-RingZahlenbereichMultiplikationsoperatorFormale GrammatikRingnetzKonfigurationsraumSelbstrepräsentationSyntaktische AnalyseCoxeter-GruppeMathematikAbstrakter SyntaxbaumArithmetischer AusdruckArithmetisches MittelBimodulMusterspracheFolge <Mathematik>ImplementierungSoftwareentwicklerSpeicherabzugDeskriptive StatistikStreaming <Kommunikationstechnik>LastSoftwaretestWort <Informatik>Rechter WinkelComputerspielWeb SiteCASE <Informatik>AbstraktionsebeneAbstrakte SyntaxInstantiierungXML
08:41
Abstrakter SyntaxbaumNichtlinearer OperatorFormale GrammatikQuellcodeMereologieRegulärer Ausdruck <Textverarbeitung>SolitärspielParserElektronische PublikationNichtlinearer OperatorArithmetischer AusdruckParserFormale GrammatikSummierbarkeitSchlussregelVerschiebungsoperatorElektronische PublikationBefehl <Informatik>Äußere Algebra eines ModulsMereologieMomentenproblemRekursive FunktionSpezielle unitäre GruppeDatenkompressionBitZahlenbereichMinkowski-MetrikSummenregelTypentheorieAbstrakter SyntaxbaumEinfache GenauigkeitOSS <Rechnernetz>ProgrammierungMultiplikationsoperatorSystemaufrufKonfiguration <Informatik>PunktUnendlichkeitMathematikMatchingGraphfärbungSyntaktische AnalyseFormale SpracheRingnetzProgrammierspracheProzess <Informatik>Hilfesystem
14:09
Formale GrammatikGruppenoperationAbstrakter SyntaxbaumQuellcodeSystemaufrufATMElektronische PublikationParserModemMenütechnikFunktionalBinärcodeBitArithmetischer AusdruckMereologieAbstrakter SyntaxbaumSummenregelOSS <Rechnernetz>Rechter WinkelRingnetzKonfiguration <Informatik>DatenkompressionDatenstrukturElektronische PublikationKonfigurationsraumNichtlinearer OperatorFormale GrammatikObjekt <Kategorie>Klasse <Mathematik>ParserDatenfeldSyntaxbaumFormale SpracheBinärdatenGruppenoperationZahlenbereichMailing-ListeAbstrakte SyntaxSchlussregelGeradeMatchingSystemaufrufAbstraktionsebene
17:22
Abstrakter SyntaxbaumQuellcodeMereologieCompilerLoopLeistungsbewertungNichtlinearer OperatorEindeutigkeitBefehlscodeByte-CodeCompilerAbstrakter SyntaxbaumZahlenbereichFunktionalFormale SpracheMereologieDatenkompressionBitZwischenspracheMailing-ListeFolge <Mathematik>OSS <Rechnernetz>LoopNichtlinearer OperatorLeistungsbewertungBildschirmmaskeRechter WinkelParametersystemBefehlscodeKonstanteComputeranimation
18:41
QuellcodeGammafunktionElektronische PublikationCompilerBefehlscodeMAPCASE <Informatik>ParametersystemBefehl <Informatik>KonstanteZahlenbereichFunktionalOSS <Rechnernetz>DatenkompressionByte-CodeBefehlscodeNichtlinearer OperatorKeller <Informatik>SystemaufrufBinärcodeArithmetischer AusdruckBinärdatenCompilerSoundverarbeitungHyperbelverfahrenTypentheorieElektronische PublikationRuhmasseWeb SiteWechselsprungRechter WinkelPhysikalismusOrdnung <Mathematik>MusterspracheProgramm/Quellcode
21:01
Abstrakter SyntaxbaumQuellcodeLeistungsbewertungLoopMereologieByte-CodeLeistungsbewertungMailing-ListeBewertungstheorieVerzeichnisdienstCompilerGesetz <Physik>Befehl <Informatik>LoopDatenkompressionCASE <Informatik>Nichtlinearer OperatorBefehlscodeRuhmasseByte-CodeComputeranimation
21:30
SpieltheorieElektronische PublikationKeller <Informatik>W3C-StandardManufacturing Execution SystemNichtlinearer OperatorLeistungsbewertungFunktionalKeller <Informatik>MakrobefehlRechter WinkelOSS <Rechnernetz>PunktAusnahmebehandlungÄhnlichkeitsgeometrieZahlenbereichBefehlscodeDatenkompressionResultanteInstantiierungBinärcodeParametersystemCASE <Informatik>SystemaufrufTermData MiningWeb SiteZweiQuick-SortBus <Informatik>
24:32
QuellcodeAbstrakter SyntaxbaumCompilerVersionsverwaltungDatenkompressionNichtlinearer OperatorCompilerImplementierungQuick-SortElektronischer ProgrammführerMultiplikationsoperatorFramework <Informatik>BildschirmfensterComputeranimation
25:10
QuellcodeFehlermeldungByte-CodeVersionsverwaltungElektronischer ProgrammführerQuellcodeFramework <Informatik>Byte-CodeRechenschieberDatenkompressionVersionsverwaltungMultiplikationsoperatorBildschirmfensterComputeranimation
25:34
E-MailComputeranimation
26:00
Twitter <Softwareplattform>COMNichtlinearer OperatorRechenschieberQuellcodeSymboltabelleToken-RingComputeranimationVorlesung/Konferenz
26:37
Nichtlinearer OperatorToken-RingFunktion <Mathematik>Formale GrammatikFolge <Mathematik>Vorlesung/Konferenz
27:05
CompilerQuellcodeFehlermeldungVersionsverwaltungByte-CodeBitfehlerhäufigkeitCOMTwitter <Softwareplattform>Nichtlinearer OperatorRechenschieberByte-CodeLoopLeistungsbewertungMultiplikationsoperatorsinc-FunktionComputeranimationVorlesung/Konferenz
27:30
Virtuelle MaschineByte-CodeGlobale OptimierungWeg <Topologie>NeuroinformatikDatenkompressionAdditionVorlesung/Konferenz
28:17
QuellcodeElektronische PublikationByte-CodeBefehlscodeElektronische PublikationZahlenbereichVersionsverwaltungDatenkompressionByte-CodeMathematikMultiplikationsoperatorVorlesung/KonferenzComputeranimation
29:15
UnrundheitVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:06
Yeah, thank you very much for that introduction So you've made it we're nearly at the end of Europe. I think I'm so glad that are still people here I noticed some people going off to the airport already, so I'm really happy that you're all here So thank you very much for joining my talk. I won't do an anecdote. I think you've heard enough during the conference
00:24
I'll just dive into my talk. I'll first briefly introduce myself Who am I and then we're going to talk about the actual substance so my name is Sebastian. I'm from the Netherlands I live near the Hague I love it there and I work for a company called ordina and more specifically the ordina python ears
00:41
Which is a smaller practice within our company that really focuses on Python development I'm also one of the code Smith's which means that I get some time for innovation for conferences being here But also just to explore Python with all the other Python developers at my company And at other companies as well just to have a little bit of fun with Python
01:00
I think that's really important to have fun with Python to really Dive into it and make it your passion at least for me. That's really what I love in my spare time I'm one of the volunteers for euro Python here this year. I was the finite lead for the financial aid program And I also did some session sharing was really interesting I'm really happy that we've made it so far into the conference
01:22
And I'm also one of the founders of Python discord Which is a larger online Python community with a lot of teenagers But also older folks who are trying to learn Python discuss Python and do other stuff with Python So definitely check it out if you haven't heard about it. Anyway, that's enough about me. Let's talk about Python So what are we going to do today?
01:42
Well, basically we're going to make a journey a journey all the way from the source code of Python to the execution and For me, it's kind of like writing magic spells I used to play games like D&D and stuff like that and with programming languages you get to write something and then something happens Often it's not what I want, but at least something happens
02:02
So that's what we're going to see today. I'm going to take you on a journey I'm going to show you how Python gets from the source code all the way to the magic all the way to the execution I'm not just going to give you a lecture about that. I think that would be rather boring But I'm going to show you how to implement a new operator in Python and it will naturally take us all the way
02:22
From that source code to that execution. However, do note I have a serious disclaimer for this talk We have time constraints, so I'm going to omit things there will be blatant emissions there will be gross oversimplifications I will take shortcuts the Implementation will not be ideal, but hopefully it will give you an overview of what happens within in the Python internals
02:45
alright If you do like to know more details if you're interested after this talk Definitely check out the Python developers guide on devguide.python.org It contains a lot of information about how you can mess with the Python internals how to compile it
03:00
But it also has excellent explanations about the parser grammar and everything else and then obviously there's Anthony shaw's book See Python internals published by real Python. It's a great book. Definitely check it out if you're interested in this Alright, so what is our journey today? Well, we're going to go from source code to execution basically in two parts in the first part
03:24
We're going to look at the tokenizer and the parser to create something called an abstract syntax tree And then in the second part, we're going to take that abstract syntax tree and go all the way to the magic using the compiler in the evaluation loop by the way If you want to check out the slides or the source code for the the version of Python that we're creating today
03:45
Check out the github repository. Everything's there couple of versions of the slides one version of the source code, but do note It's an educational implementation Don't use it in production. I'm not maintaining it. It's a crappy implementation. Please don't use it for anything serious
04:01
Alright, what is a pipe operator? Well, the pipe operator is something that's part of a lot of languages But not Python and it's basically another way of calling functions So say you have a function that takes one argument like this double over here Just takes a number and it multiplies the number by two and it returns to value Well, the pipe operator allows you to call this function in a different way
04:23
You provide a value on the right-hand side of the operator Then you use the operator itself Then you name the function and then the argument will be passed into the function and you get to result out of it Well, this is obviously not really interesting. It's just like calling the function double one But what you can do with this is you can build pipelines
04:42
So you can start with a single value Then you can pipe it into various functions to process it you get something out the other end So in this case, this would be equivalent to calling double one and then insert that into another double call the nested function call So this is what we're going to do today Well, just to be clear. This is not a part of Python
05:02
I don't think it will ever be a part of Python there were a few proposals for this They were rejected for good reasons. Maybe in the future who knows ask a core developer not me And the implementation is purely educational. I've already mentioned that So the first part we want to get from the source code
05:21
So the characters on the right-hand side all the way to that abstract syntax tree that you see there So this tree like representation of your source code So if you have a single Python file with this line in it only this line You'll get something like you see there the tree representation of your source code. You have a module a single expression
05:43
It's a binary operation an operation with two operands and value to the right-hand side and a value to the left-hand side And that binary operation has a constant a 10 It has the operator itself the coal pipe and then it loads a name the name of the function Obviously this won't work in a file and so on because there is no function here
06:03
But this is the general idea of what we're trying to do in the first part We're trying to get from that part to that nice tree over there So first look at our source code We humans we are very good in reading we recognize patterns We immediately see that we have a 10 that we have a name double that we have a weird operator in the middle
06:25
That is unfamiliar in Python, but at least we immediately recognize bits But if you think about it for Python at the start, this is just a stream of individual characters There are 12 characters here and they make up our source code and Python has to understand those characters
06:42
and the first step Python fakes is Basically extracting the tokens from this source code and tokens are the minimal parts that still have a meaning So for instance, we have a number token over here If you split this up even further into a 1 and a 0 you lose the meaning of the number 10
07:00
We have a name over here. Obviously if you remove a character, it's no longer the same name So this is a name token and the first thing that we need to implement is that Python will recognize this Token here in the middle this operator this two character operator as one single token This isn't very difficult to do. This is just a configuration in a file. If you look in the Python repository
07:24
There's a file grammar slash tokens, and it's just a big mapping of token names to token character sequences So all that we have to do is we have to add our own token into this file and There's no meaning here yet. This is just a description of a token
07:40
So I've called it V bar greater filled a vertical bar greater and I've mapped it to the token sequence that we want to introduce This isn't All of it, but what we still need to do is regenerate the tokenizer itself the actual Piece of Python that will get this tokens from your source code
08:01
Because Python will not read this file every time it tries to parse source code for that We just have to run some commands and this will be a recurring theme in this presentation Change something run a command and then you can see the result So these are the commands I'm not going to go into detail in all those in all these commands because you can find them on the internet
08:21
You're not going to remember them here anyway But after we've done this Python will now be able to recognize our new token our Vbar greater token and that's already a big step, but it doesn't really know what to do with that token So there's no grammar rule that tells Python if you see this token then this has to happen
08:41
So the next thing that we will need to do is we need to add support for this new token in Python sparser by adding it to Python's grammar Well, the grammar also has to produce something which is the abstract syntax tree that we saw earlier So we also have to tell the grammar how to generate the abstract syntax tree and make sure that there's something for our coalpipe
09:03
available in that abstract syntax tree So let's look at our grammar yet So since Python three points Oh, I forgot three point nine the new pack parser has been introduced and this has a completely new grammar It's very flexible. It's very powerful But we first have to look into the syntax of this grammar just a little bit to add our own rule
09:26
So I'm just going to take a very small dive into the parson expression grammar that Python uses just enough to know to define our new rule for the coalpipe operator So if you imagine a very simple programming language, and it only has two expression types
09:44
It has a sum and it has something called an atom And in this very simple language, there are only two grammar rules The sum and the atom there might be some other four statements and stuff like that But we we're going to ignore that for now We're going to focus on these two then this is basically what you can define impact parser
10:02
This is a little bit simplified But these could be grammar rules and just to color code them grammar rules can reference each other and this is how? Python goes down all the grammar rules to see what it matches So it's first try it starts by trying to match the sum rule and Within the sum rule there are options that reference the Adam rule
10:22
Which is just a number and that is how we ever consider the Adam rule, so if we just have this single piece of Source code it's just a number it will first try to to match the sum rule it has two alternatives That's what the vertical bars mean it has an atom plus an atom and it has an atom on its own
10:43
So this matches the second alternative in the sum rule Which is an atom then we look at the atom because it's just a number so we can parse this So in this very simple grammar example we can parse a number on its own How what about this one on this one? We have an atom a number plus and then another atom which matches the first rule of the Sun
11:06
So it's very easy to see that this grammar rule will now be able to match this simple piece of grammar as well But now what about this one? Think about it for a moment Are our grammar rules able to parse this very simple statement I?
11:25
See some people shaking yes other shaking no well the problem here is that when you start parsing your expression You will consume these parts we can Parse this very easily and what we're done left with is a plus and a three And we have no rule that matches a plus and a three on its own
11:42
So how are we going to match something with two pluses? Obviously we can add another rule But what if we want an atom plus an atom plus an atom plus another atom do we have to add another rule? If you want four pluses or five or six or if we want an infinite pluses We'd be busy quite a long time who wanted to add alternatives for all those different scenarios
12:05
but there's a very simple solution for that and You're probably going to love this It's just recursion so what we can do is we can change the first Alternative in the sum rule we can make it reference itself And now we can have sums that are embedded in other sums embedded in other sums
12:23
And just remember the sum can also just be an atom on its own so we can still match an atom plus an atom but now we can also match a sum contained in another sum and This is basically all that you need to do to get an arbitrary number of operators in a row And if you think about it, this is precisely what we need for our new pipe operator
12:45
We also want to be able to build an arbitrary long pipeline So the grammar rule that we will need to add is a grammar rule that has such a recursive relationship So let's see that this is the existing grammar file in Python grammar slash Python gram
13:01
Here you see the shift expression if you want to do bit shifting and you see the sum expression I'm just going to insert our new grammar rule between those two, so let's make some space This is probably isn't the best place to insert it, but it means that it's very easy to insert it So that's why I've chosen it So here we're going to add a new rule. Let's call it pipe and as you can see here
13:24
This is just our recursive relationship There's just one problem our grammar rule isn't Referenced by any other grammar rule in Python. So the parser will never consider it when parsing source code So what do we have to do? Well, if you look at the shift expression, it actually references the sum expression
13:43
That's how the grammar rules flow down So the only thing that we need to do to insert it We have to change the references in the shift expression to the pipe one and in the pipe Can then reference to someone so if we change this? The shift expression now references to pipe one the pipe to someone and now our grammar rules can flow down again
14:02
And this is basically the only thing that we need to do to add our new grammar rule for our new pipe call operator So now we can parse this right? We're done Well, not quite But because we also still need to be able to create the the tree structure that you see there on the right
14:20
We have to be able to fit it into our abstract syntax tree So, how are we going to do that? In the old parser there used to be an intermediate step the concrete syntax tree But with the new pack parser, we don't need that anymore And the reason why is that because we now have something called grammar actions if you look here. This is a grammar action
14:42
Between the curly braces and because we're targeting C Python This is basically just a piece of C code that is embedded into the grammar file So whenever Python matches a some rule it will then call this C function To create the classes that we see there to create a tree tree structure that we see there
15:01
Well to create a binary operation tree structure. We need the right-hand side of the operator So the value we need we need the left-hand side and we need the operator So that's the information that we're going to pass into this function and that we can do by assigning names To the parts that we match in our expression So we match in a to the right hand side pass it into the function
15:22
We match a B to the left hand side pass it into the function We know that this is an add operation because this is the sum rule so we can hard code the add operator And there are some extra bits having to do with line number and stuff like that We're going to ignore that for now, but they're very handy for trace backs and other kinds of interesting things
15:40
So this is all we need to create a binary operation with an add operator Can we now do that for our own rule? Obviously we can just copy the approach because our new operator is also a binary operation So we can just make it call the same function But instead of using the add operator we can use the coal pipe operator here
16:01
Just one tiny problem The AST uses classes and there is no class called coal pipe yet It just doesn't exist yet So we have to create that so that we can actually build this abstract syntax tree with that coal pipe node Somewhere in the middle for the operator field. So how do we do that? Is it difficult? Do we have to code a lot?
16:22
Luckily not because this is another configuration file This is in parser slash Python dot a SDL which stands for abstract syntax definition language And as you can see here somewhere in the middle There's a part for the operator and here's our add option just in an option list And this is all you need to do to have a generator create classes for you
16:46
So what do we have to do? Just add our new operator at the end as an other option Then we regenerate the AST we run another command and now our classes will be created for us So this is all we have to do to add support in the AST
17:02
Now we can regenerate the entire parser and now we are actually able to parse Expressions with the coal pipe operator in it You can see here that we have a binary bin of notes somewhere in our AST tree and it actually uses a coal pipe class Object in there to represent the operator. So this is all that we need to do for the new grammar
17:23
And now we're done now we can go from source code to an abstract syntax tree But this is all fairly static. Nothing happens yet So to do that, we need to move on to part two where we actually going to Where we actually going to transform this into something we can run and then actually execute it
17:41
So in part two, we will look at the compiler So if you've ever joined an online discussion about is Python a compiled language or not, please don't they're all very toxic But we are going to compile this into some form of an intermediate language We're going to compile this into a little bit of bytecode And to do that
18:00
We need to have a bytecode for a new operator because bytecode is just a long list of instructions for Python Long sequence of bytes a long sequence of numbers and each instruction that we're going to execute Has its own number or its own byte and we need one for our new operation And obviously there is already a bytecode for calling functions
18:21
But I'm going to ignore that one because it isn't fun to use what's already in Python So we're going to add support to the compiler by creating our own instruction then we're going to make the compiler actually use that instruction write it into the bytecode and then we're going to To add support in the evaluation loop to actually do something with that code
18:42
So let's do that. This is the only fight Python file that we'll see today We can use it to define our operation codes our op codes Our op code doesn't have an argument. Don't worry about that for now But it means that it has to have a number lower than the half argument constant. So I'm just going to add it here I'm going to call it binary pipe call
19:01
That's for us so that we can understand it and the bytecode and the number that we associate with the operation is 90 And I have to increase all the numbers below it. That's a tedious job, but we'll have to do it Now we can regenerate all the op codes Do you see the pattern yet? And now the op code will actually be generated for us
19:22
Right now we have an op codes now We still need to make the compiler actually write that op code into the byte code. So how are we going to do that? Well, the compiler is just going to visit all the nodes in our AST and it has functions that handle that visit and Somewhere in our compiler. There's a function called compiler visits
19:43
Expair one expression one which eventually gets called to handle expressions We don't actually have to change it But there's an important fact here if you look at this function It's just a massive switch case operations and for each type of expression including the binary be bin up kind
20:00
There's a special case and here is our case We first are going to visit the left hand side of the operator Which is really important because we need that value write all the instructions for it then we're going to visit the right hand side write all the instructions to Evaluate the right-hand side and then we're going to add the operation for the operator and
20:21
To know which op codes to write there's a helper function bin up and this is the function that we actually need to change This is another switch case statement This will get an AST node like this add here and it will return the bin up that we actually want to execute or Add to the byte code. So we need to add another case
20:41
Write our binary pipe call operator and now we can write it into the byte code There's one final thing that we need to change in the compiler. It's the stack effect Python uses a value stack You'll see that later And the effect of our operation is that our value stack decreases by one value, but don't worry about that for now
21:02
And this is the compiler and now we can go from an AST To a long list of instructions and this is all we need to write our new operator our new op code into the byte code And now we get to the evaluation loop and this is really where the magic happens The evaluation loop like the name says is just one giant loop that goes around and around and around and around
21:25
executing all the instructions in the byte code and Inside of that loop. There is a massive switch case statement and I'm not Exaggerating, it's really massive. Look at the source code in Python slice She evil dot C and for each opcode it has a case
21:42
So for instance here is the binary subtract and this is the code that actually gets executed whenever it sees such an opcode for a binary subtract something minus something else But here we have a problem Because how do we get the values that were to the right and to the left of the operator?
22:00
How do we get them back to actually process them here? Well, this is why Python uses a value stack And from that value stack we can get the values that we need to perform the operation How does that work say that we have this simple expression for minus one? What we saw earlier is that Python will first write the instructions to evaluate the left-hand side when it's done
22:25
It will put that value onto the value stack To remember it Then it will evaluate the right hand side and it will put it on to the value stack and the value stack is really a Stack it will put a three on top of the four on the value stack just so that we're able to use them later
22:43
So when we enter this new function for the operator the three and four are on our value stack so now that we can use the pop macro just to get the three out of the value stack and Make right point to that value For left we do something similar. We leave it on the value stack, but we make left refer to that value
23:03
That's there on the value stack Now we have our values we can call a C API function by number subtract Passing the two values we get a result out of it result will now point to the one We are now done with those values We can decrease the reference counts for the values and then at the end we need to do something with the result with the resulting
23:25
Value. So what do we do? We just put it back on the value stack. We're going to replace the value That was already there So after performing the operation There's one less value on the value stack and that was to minus one that we saw earlier Then there's some error handling and then we do a dispatch to say to the evaluation. Please move on to the next value
23:46
So how do we do that for a binary pipe call? Oh, you just copy paste this code because this does basically what we need We change the targets to our binary pipe call operator We get the values like we did before but now we don't want to subtract them. We want to call a function with a value
24:03
There's a very handy C API function in Python 3.9. It's not in the long term stable one So it might be deprecated but it's very handy here. And here you can call a function with one argument And remember the function was on the right hand side So we pass in function right first the value was on the left hand side
24:23
so we pass it in second we get the result decrease the reference counts and we Put the resulting value back on the values value stack and this is all we need to do to get from the AST to the magic and Now we've completed our journey
24:41
We can go all the way from source code to execution and all the steps that are in between There's just one thing left to do let's compile our new version of Python and if you're waiting for it You can you might as well just have a sort fight and after that's done You can run your new version of Python and you can use your operator. It will actually
25:01
Work if you follow these steps, there are some quirks especially in operator precedence with this implementation but it will work in principle and Basically, that's it. We've seen a lot of Python internals in a short time. Don't worry about it This is just for a framework check out the books to dev guide source source code and slides are available
25:21
And if you get weird errors, try running make clean or clean all on windows And look into something called the magic number because we've changed the bytecode version So we need to change the bytecode version in C Python as well Before we go, I hope you've been you've enjoyed your a Python It's been great for me, but we also need to organize the next edition in
25:44
2023 I don't know when or where it's going to be. But if you want to be a volunteer just like me Join us to help organize Python 23 you can send an email there. You can just talk to me after my talk and that's it I hope you've enjoyed my talk. Thank you very much
26:02
Thank you very much, thank you Wow, wow. Wow, I expect the only way we're gonna see a lot of forks of Python nowadays So do we have questions so when you'll be ready just queue up and we have we can okay go on
26:24
Just something silly in the beginning when you created the tokenizer You had a symbol there. Did you use that afterwards? I don't remember the symbol the vertical bar and yeah the V bar Greater well, we don't use the name that that we can use at the tokenizer output
26:45
But we haven't used a name in the grammar So we haven't used a name but the token sequence is obviously our new operator. So that's the actual operator that we use So so why did we give it a name because we had mostly for us humans to make it readable
27:05
So, yeah Another one, please. Thank you for the talk. I really enjoyed it Like stupid question like every time I'm here that like the bytecode evaluation just a loop looking at the top of the stack It seems so like limiting like you have just like one of you while you can look at
27:25
How does it happen that it like us like normal performance like it doesn't yeah It's very interesting. So there are obviously a lot of Optimization since there but basically this is just kind of like a virtual machine you have instructions you have a value stack that you keep track of some kind of register and and that's
27:42
Really powerful you can really do a lot with it. So maybe in the advent of code, which is an online puzzle I think in the 2019 edition you will actually Build your own kind of like in scope computer using your own kind of bytecode and targets So I recommend you to check it out and you can really see how powerful it can be. So
28:04
I'm not quite sure how Performance you can get it but Fun yeah, thanks So, um, I don't see anyone else can I have a question though? Yeah, so
28:20
One minute to describe what's the magic number the magic number? So I actually had a file about this So whenever Python compiles your code, it will write to a pi C file You've probably seen it somewhere and that bytecode is versioned It has all the instruction numbers in there But if you're going to change the instruction numbers and obviously all the old byte codes, they're no longer valid
28:43
They use the old numbers. So there's a kind of a versioning in Python It's called the magic number and it will version that bytecode You can actually change that magic number if you change the the bytecode opcodes So that all the old pi C files will be marked as how do you say that stale and it will recompile all your Python
29:03
So just to avoid very weird things happening with old byte codes with changed byte codes. It gets a mess So that's what that's the magic number. Yeah. Thanks Well, thank you. And this is the only time we have so please Give a round of applause now. Thank you very much