Code reloading techniques in Python
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Untertitel |
| |
| Serientitel | ||
| Anzahl der Teile | 542 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 2.0 Belgien: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/62035 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
FOSDEM 202373 / 542
2
5
10
14
15
16
22
24
27
29
31
36
43
48
56
63
74
78
83
87
89
95
96
99
104
106
107
117
119
121
122
125
126
128
130
132
134
135
136
141
143
146
148
152
155
157
159
161
165
166
168
170
173
176
180
181
185
191
194
196
197
198
199
206
207
209
210
211
212
216
219
220
227
228
229
231
232
233
236
250
252
256
258
260
263
264
267
271
273
275
276
278
282
286
292
293
298
299
300
302
312
316
321
322
324
339
341
342
343
344
351
352
354
355
356
357
359
369
370
372
373
376
378
379
380
382
383
387
390
394
395
401
405
406
410
411
413
415
416
421
426
430
437
438
440
441
443
444
445
446
448
449
450
451
458
464
468
472
475
476
479
481
493
494
498
499
502
509
513
516
517
520
522
524
525
531
534
535
537
538
541
00:00
MaschinenspracheComputeranimation
00:39
Software EngineeringEreignishorizontSocketLambda-KalkülZeichenketteVektorpotenzialSocket-SchnittstelleZeitzoneFramework <Informatik>SkriptspracheElektronische PublikationWeb-SeiteProzess <Informatik>VersionsverwaltungTermQuellcodePatch <Software>MaschinensprachePatch <Software>Elektronische PublikationProzess <Informatik>AggregatzustandEinsProgrammierumgebungMathematikVariableMereologieFramework <Informatik>InternetworkingKategorie <Mathematik>QuellcodeMaschinencodeBenutzerbeteiligungTermInterpretiererProgrammierungSocketComputeranimation
03:11
FreewareProzess <Informatik>AchtMaschinenspracheStörungstheorieGamecontrollerGanze FunktionFramework <Informatik>DatenbankBenutzerbeteiligungAggregatzustandComputeranimation
03:47
ThreadMechanismus-Design-TheoriePhysikalisches SystemEreignishorizontSystemprogrammierungElektronische PublikationVerzeichnisdienstp-BlockEin-AusgabeLuenberger-BeobachterHochdruckProzess <Informatik>ROM <Informatik>IRIS-TMaschinenspracheProzess <Informatik>Kartesische KoordinatenProgrammierungBenutzerbeteiligungBitThreadEreignishorizontElektronische PublikationDifferenzenrechnungNabel <Mathematik>StörungstheorieServerEinfach zusammenhängender RaumAggregatzustandMereologieObjekt <Kategorie>MathematikDateiverwaltungPhysikalisches SystemSocketDatenverwaltungMomentenproblemPortabilitätQuellcodeInterface <Schaltung>ComputerspielProgrammbibliothekComputeranimation
07:19
MaschinencodeDebuggingAppletProtokoll <Datenverarbeitungssystem>VersionsverwaltungProgrammbibliothekBimodulHochdruckLesen <Datenverarbeitung>Proxy ServerFunktion <Mathematik>QuellcodeMinkowski-MetrikAttributierte GrammatikProgrammierumgebungSummierbarkeitInstantiierungKlasse <Mathematik>Objekt <Kategorie>MaschinenspracheSpeicherbereinigungHalbleiterspeicherProxy ServerRegulärer GraphZählenEin-AusgabeComputersicherheitGamecontrollerArithmetischer AusdruckTypentheorieVariableMultiplikationsoperatorParametersystemBimodulFunktionalResultanteQuellcodeStörungstheorieSystemaufrufDebuggingVersionsverwaltungZeichenketteSoftwareschwachstelleAppletNamensraumPatch <Software>ProgrammbibliothekMaschinencodeElektronische PublikationDatenfeldDemo <Programm>SkriptspracheEinfach zusammenhängender RaumKategorie <Mathematik>ZahlenbereichLeistung <Physik>Exogene VariableByte-CodeAggregatzustandVererbungshierarchieInterface <Schaltung>DatenbankMereologieZweiNebenbedingungFormale SpracheComputeranimation
17:11
DatensichtgerätSpieltheorieFlächeninhaltW3C-StandardOvalE-MailGammafunktionFunktionalProgrammierungAggregatzustandVorzeichen <Mathematik>MultiplikationsoperatorMaschinensprache
18:36
MaschinencodeAggregatzustandStörungstheorieURLDatenbankMaschinenspracheRechenschieberBenutzerbeteiligungComputeranimation
19:08
Computeranimation
20:10
GammafunktionMIDI <Musikelektronik>KreisringW3C-StandardOvalE-MailLeistungsbewertungMaschinenspracheBefehlscodeNabel <Mathematik>ProgrammbibliothekInstallation <Informatik>
22:02
Flussdiagramm
Transkript: Englisch(automatisch erzeugt)
00:05
Please welcome Hugo, who's gonna do the next talk on code reloading techniques in Python. Hello everybody, and I'm glad to speak to you, and I hope I will have a good presentation,
00:28
you will be happy about replacing Jonathan's talk, who couldn't make it today. So we'll be talking about hot code reloading in Python, and a few different methods to do this. Short introduction, so I discovered Python in Linux in 2003, and I think I've attended
00:46
almost every FOSDEM since 2004, and I really, I think FOSDEM made me, and I'm really happy to be speaking here, to be here with you. In Python there are a lot of things that we can do, and I'm really amazed by all
01:02
the flexibility of Python and the access to the internals, and when I started discovering Python I was 16 years old, and I discovered you could play with exec and socket and combine them and make trojan horse, and I found that really cool. Many years later I discovered people were basically taking the same idea to make something
01:24
called Jupyter notebook, you may have heard about, something where you receive Python code over the internet and then you just execute it somewhere. And I really like tinkering with Python, I also created my own web framework before
01:41
they were Django and the most popular ones that are more mature now, and one thing I discovered is you could use exec file to execute code that's in a Python file in a specific environment. So you have these environment variables in your Python code, and you can override them, so when you execute code from a Python file in a specific environment, you pass the environment
02:06
in the dictionary above here, it can only access these variables, so you could restrict the access to imports, the access to a lot of things, but Python is Python and there are always ways to escape this by accessing the properties of objects, so there is no safe
02:23
way to run Python codes contained, like you could use Lua or something, when you have Python code it always has access to everything that the C Python interpreter has access to.
02:40
And so I will be talking about code reloading, which is the process of replacing part of a program with a new version. It focuses on changes in the source code, a term mostly used for interpreted languages, and you have two ways to do code reloading, you have cold code reloading, which means that you are stopping the process and you are restarting it, and you have hot code reloading
03:04
where you keep the process running and you just patch it while it's running, keeping the state. A small analogy, cold code reloading, you really have to, you can really switch anything you want, but it takes a while, hot code reloading, you just do it while it's going
03:21
on. Cold code reloading is the most popular technique for code reloading in Python, so you stop and restart the entire process, and it's easy, reliable, but you lose state, and getting it back might take a while, it's very popular in web frameworks because all
03:42
of the state is stored in databases. And you also have the easiest way to do code reloading that you all used, control C, up enter, I still end up using it so often, and there is tooling to do that, if you
04:01
look at how Django and Flask do it, you have good examples, the Django source code especially is very easy to understand, very clear, and the main part of it ends up being this part here, so it has this object called a reloader, and will run this function, and
04:22
it will basically run your main app, your main process, in a thread, and when there is something that has changed, it looks at changes in the file system, when it has changed, it will restart your program in that thread, so it's basically running all of Django in a thread, and that allows it to stop it, reimport everything, and then
04:44
restart that part of the process, so you still have one process running, only one process, but it's running all of your application in a background thread that's managed by this reloader object, that's what happens when you do a Python manage.py run server using Django, Flask uses something very similar, the code is a bit easier to read when
05:04
you look at Django, so really have a look at this autoreload from Django, it's a really good example on how to do things, and under the hood, there are different ways of detecting that a file has changed, of course you could just manually, every second or every
05:21
few second looks if there are changes on the file system from a thread again, but there are ways on different operating systems to have direct notifications, direct signals when something is happening. On Linux you have the inotify API provided by the kernel, that basically gives events so any process can look for changes on files, and the very
05:45
simple way to use it in a shell is just to use this inotify to wait, so you can just say I'm waiting for anything to happen in this directory, and then when something happens you just do something. In Python you have the py inotify binding to this inotify
06:00
library, and you have watchdog, and watchdog is a multiplatform library in Python to do exactly this, so it works on both Linux, MacOS, Windows, BSDs, not micropython, and the interface is quite simple, a bit of a classy overhead, but you can basically schedule
06:25
an observer and start it and it will use callbacks when something is modified, so you can do whatever you want to do at that moment. Hot code reloading, that's something that's less common, especially in Python, but in general it's less common, it has its drawbacks
06:45
because of the states, it can lead to inconsistent states, but it also has advantages because sometimes this state that you're interested in, you're making a video game and you have an NPC, and to test it you have to go back to that same space, recreate the whole
07:01
behavior, it's hard to tweak the behavior live when you have sometimes a web socket or socket connections that you have to keep open, when starting your program takes a while because you're initializing a bunch of things, then it can be quite useful to do hot code reloading. And there are two challenges if you want to do this, one of
07:24
the challenge is to find and load the new code and then you need to replace the references where that code is used in the code, in memory. And it's not a new thing, other languages do it, Java especially, they have a way to use the debugger and you can replace
07:44
a version of a class file which is the PYC of Java, the compiled file of Java, you can replace it from your IDE directly while it's running using the debugger, the only constraint is that the interface of the class, the methods, the properties of that class
08:01
may not change. In C++ you can do hot code reloading by using DLLs, the dynamic link libraries or shared libraries and that allows you to also do hot code reloading, so it's possible to replace DLLs and do hot code reloading in C++ as well.
08:22
If you want to do it in pure Python there are three main ways to execute code from Python, you have the import module, if you just import something it will basically execute everything in that file, if you were at the micro Python talk, sometimes you just import blink and it just runs it, that's a weird feature of Python but it basically
08:44
just executes everything there. You can use exec, this function that takes a string and then it will just parse the Python code from that string, execute it, potentially in a specific environment you can add specific variables to the environment of this execution
09:01
and you have eval, that's basically evaluating a Python expression and getting the result out of it. So if you want to make a small shell or ask for a user input and you use eval, the user can type any Python code he wants, which could lead to security issues if that's untrusted input, but it can be quite nice if you want the user to be
09:22
able to use powers of numbers with two stars and any Python supported expression. So if you want to use the import base, if you use import, something that you can
09:40
use is this imported reload and you can reload a Python module that you imported. So if you just have some code that's importing a module, you can reload that module, but all the references you have will still be to, if you have a reference or a function inside that module, it will still be a reference to the old function, to the bytecode
10:02
of the old function, not the updated version. However, if your module is named module and you have module.my function and you always call it module.my function, then when you reload it, if it can access the new reference to the new loaded module, then this will work. So it's quite an easy way to do reloading of codes if all your references
10:26
are not inside the module but reference always through the whole module.my function. One issue you have if you use this import base reloading is sometimes people write
10:45
code in the middle of the Python file. You just have a small script that's containing a while through or you have a connection to a database that takes 10 seconds and that's blocking and every time you're reloading, this will be in your way every time and will
11:01
slow you down. Another way you can do Python code reloading is to just take control yourself of the execution of the refreshing. So if you have some Python code here, you can execute the code above within the environment of this module, module.py.
11:25
So if I have a function my sum in module.py, by running this I can basically refresh it and say okay, now I want to replace module.my sum with the new function, the new code here. And so if you want to wrap everything together, what can be quite interesting
11:48
is to have a proxy. So instead of manipulating directly the function you're interested in, when you want to be able to reload something, you don't pass around a reference to that function but to a proxy that can change its own reference. That way you can hot reload
12:03
something and then everything that's using it is not using my function directly, it's using a proxy to my function. And you can override the call attribute. So here you have this proxy here and it's a proxy to work in progress. It keeps this reference to
12:21
the function and then every time you call the proxy, so the instance of this proxy, it will basically just forward every argument to the function and return the result. So you can just proxy a call to a function this way with a mutable reference to the function here. This mutable reference you can change later. You can do something similar with classes.
12:47
So if you look at a Python class, an instance of a class in Python, it basically has two very important attributes, class and dict. Class is a reference to the class that it
13:00
inherits from. So if you instantiate a class, every instance has a reference to the class it instantiates and that's referenced because Python is mutable so you can change the underscore underscore class of your object and forward it to another class and bam, your object is now from another class. I see some people don't like this idea. It's Python with great
13:29
powers and great responsibility. You can do very nice things in Python. Please do not unless it really makes sense. And you have this underscore underscore dict object which contains all the properties that are part of your instance. So if you do my instance
13:47
dot a, it will look, okay, is a defined inside underscore underscore dict? Yes, it will just return that. No, then it will look on the class and on the parents of that class for this attribute a, attribute or method because all the methods and the attributes
14:05
are in this underscore underscore dict part. So in practice if you have this, an object, an instance and you replace the underscore underscore class and you keep the underscore underscore dict, then you are switching, reloading what class that instance is part of and then
14:25
if that new version, new class is the new version of your class, you just reloaded live the instance and the code of your class. So this allows you to basically do hot code reloading of the class of all the instances if you patch this. And then
14:46
the last thing that's useful in Python is the weak references. So weak references allow you to keep a reference to an object but still allow it to be garbage collected. So when you have a reference, the garbage collector uses a reference counting and also
15:05
a regular collection for circular import, circular references but when you keep a weak reference to something, then it will not increase that garbage counting counter so it will allow the object that's targeted to be garbage collected at some point. And so you have
15:30
this weak ref library in Python to have a weak reference to something. You can get the source codes of the new source code using the inspect module in Python.
15:43
You have inspect.get source of my function and it will just return you directly the source code as a string of your function and inspect.get source of a class does exactly the same thing so it's very easy to get access to the source code of any Python function or class.
16:02
And if you put it all together, you can say okay, I have my source code, I just get the new source code of my targets which uses inspect.get source. Then I have some environment, the environment of the module and some new environment with the variables created by
16:24
the code and I just execute the source code within the environment of the module so you can access all the imports and all the variables defined in that module and I have this local so that every variable created during this execution is available within
16:41
this name space. And I just returned the field from this dictionary here that's named the same as my targets and now I just hot code reloaded it. Putting it together, I made a small module in Python where I added this autoreload feature
17:03
so you can decorate any function and it will reload it automatically while it's running. We can have a look at a small demo here. So I have some Python code with global state and two functions that are processing something and I will just, I can just run this.
17:23
But if I want to be able to live code, I need to, I can just reload those. I'll just mark these two methods as autoreload. So there's no magic. You have to mark them
17:42
as reloadable. Too much magic is bad. There is already enough. And so here on the right side, you see that it's executing this program. I don't really like the way it's printing the value so I want something more English like value is. And this complex processing thing, it's, yeah, maybe it was a 20 I wanted. So now
18:07
it's multiplying everything by 20 or it was a negative value so it's changing sign all the time. And this is just while running, I can just swap it directly and as you can see, the program is running. It's maintaining the state but you can still change the code.
18:24
If you do a typo, then it will just show you that there is an invalid syntax and wait for you to fix it to keep running again. And you can just pip install it. It's
18:44
on pipe UI so if you end up in the situation where your state is complex to get in Python and you want to work on something, you are not doing web with all your state in a database. Have a look at reloader. It's a simple way to just hot code reload and
19:03
trick your code while it's running. You can find the slides on this URL and thanks for watching. I believe we have five minutes for questions or remarks if you didn't like
19:28
it. So if you have any questions, don't hesitate. Yes?
20:00
So you can use AST. Yeah, so what you're saying is that the normal, let me just get a normal shell. So what you said is that the normal ways of executing Python code is not safe
20:21
but in the library AST, literal evil and this one should be safe to execute, right? This only allows to evaluate constant stuff. Okay, that's good to look at. Thank you.
21:01
So there is a library that allows you to do safe evaluation by filtering the opcodes that the code is allowed to run. That's what you're saying. Okay, that's quite interesting as well. Yeah. So how do you kill it? Phone calls?
21:29
Phone tools. Oh, okay. PWN tools. Okay. I'm not sure I want to pip install something with PWN inside. Yeah, any other question or remarks? Okay, thank you.