Executing scripts in a few milliseconds with MicroPython

Video thumbnail (Frame 0) Video thumbnail (Frame 1341) Video thumbnail (Frame 2035) Video thumbnail (Frame 9676) Video thumbnail (Frame 11404) Video thumbnail (Frame 16084) Video thumbnail (Frame 16844) Video thumbnail (Frame 18464) Video thumbnail (Frame 21166) Video thumbnail (Frame 31092) Video thumbnail (Frame 32201) Video thumbnail (Frame 34110) Video thumbnail (Frame 35381)
Video in TIB AV-Portal: Executing scripts in a few milliseconds with MicroPython

Formal Metadata

Executing scripts in a few milliseconds with MicroPython
Title of Series
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Executing scripts in a few milliseconds with MicroPython EuroPython 2017 - Talk - 2017-07-14 - Arengo. Rimini, Italy Command execution time can become important in a number of applications. Commands executed in command-line completion need to execute in less then 100ms or users will perceive a delay. In Shell scripting one might want to execute commands repeatedly in a for loop and fast execution times makes this more feasible. Python is a very powerful language but has a much slower startup time compared to other interpreted languages like Perl, Lua and Bash. It can take up to 10 times longer to startup then some of these other languages. MicroPython was written as a lean implementation of Python 3 with a small subset of the standard library mainly intended to run on microcontrollers. But it happily runs on Unix systems with excellent startup performance, making it an ideal candidate for implementing certain time sensitive commands. This talk will: Explain when achieving fast execution times matters and when it doesn’t. Present two different approaches to measuring command execution time, one simple and the other more detailed and accurate. Compare execution times of a simple set of scripts that add two numbers in an number of different interpreted languages (micropython, python3, awk, perl, lua, bash). Present an example use case of MicroPython on Unix. Bash completion for pip install that completes the names of available packages live from a remote pypi mirror. Demonstrate the auto completion script with pip on a local pypi mirror
Scripting language Web application Scripting language Multiplication sign Coroutine Cartesian coordinate system
Befehlsprozessor Password MiniDisc MiniDisc Neuroinformatik
Scripting language Multiplication sign Source code Survival analysis Frustration Complete metric space Function (mathematics) Perspective (visual) Computer programming Usability Neuroinformatik Measurement Roundness (object) Hooking Single-precision floating-point format Physical system Scripting language Programming language Structural load GUI widget Complete metric space Variable (mathematics) Measurement Benchmark Type theory Exterior algebra Befehlsprozessor Process (computing) Website Software testing Right angle Text editor Cycle (graph theory) Quicksort Resultant Functional (mathematics) Game controller Implementation Computer file GUI widget Real number Letterpress printing 2 (number) Number Goodness of fit Latent heat Programmschleife Software testing Text editor Domain name Pairwise comparison Stapeldatei Standard deviation Mathematical analysis Planning Letterpress printing Line (geometry) Word Software Personal digital assistant Calculation Password Statement (computer science) Video game
Purchasing Context awareness Implementation Run time (program lifecycle phase) Multiplication sign Real number Execution unit Virtual machine Letterpress printing Microcontroller Computer programming Number Neuroinformatik Befehlsprozessor Computer hardware Computer architecture Physical system Scripting language Stapeldatei Consistency Web page Sampling (statistics) Instance (computer science) Statistics Benchmark Measurement Type theory Befehlsprozessor Process (computing) Software Right angle Pattern language Cycle (graph theory) Internet der Dinge Pressure Resultant
Server (computing) Implementation Game controller Computer file Code Multiplication sign Set (mathematics) Microcontroller Function (mathematics) Client (computing) Regular graph Number Different (Kate Ryan album) Term (mathematics) Befehlsprozessor Network socket Single-precision floating-point format Core dump Energy level Selectivity (electronic) Implementation Scripting language Programming language Stapeldatei Server (computing) Expression Graph (mathematics) Content (media) Client (computing) Core dump Line (geometry) Connected space Proof theory Befehlsprozessor Process (computing) Network socket Mixed reality Computing platform Right angle Quicksort Cycle (graph theory) Physical system Resultant Library (computing)
State observer Presentation of a group Group action Code Multiplication sign Demo (music) Source code Set (mathematics) Price index Complete metric space Parameter (computer programming) Mereology Software maintenance Usability Subset Web 2.0 Spherical cap Computer configuration Interpreter (computing) Core dump Local ring Programming language Link (knot theory) Moment (mathematics) Fitness function Electronic mailing list Physicalism Mereology Complete metric space Variable (mathematics) Statistics Benchmark Demoscene Connected space Radical (chemistry) Type theory Compilation album Quicksort Resultant Spacetime Laptop Trail Functional (mathematics) Server (computing) Implementation Perfect group Link (knot theory) Computer file Virtual machine Canonical ensemble 2 (number) Revision control Causality Term (mathematics) Form (programming) Condition number Execution unit Stapeldatei Information Physical law Expert system Usability Line (geometry) System call Frame problem Integrated development environment Revision control Interpreter (computing) Video game Natural language Local ring Freezing
Installation art Voting Demo (music) Computer configuration Equals sign Subject indexing Price index Pattern language Complete metric space Distance Graph coloring
Presentation of a group Complete metric space Mereology Computer programming
Pulse (signal processing) Presentation of a group Code Multiplication sign Workstation <Musikinstrument> Execution unit Archaeological field survey Real-time operating system Function (mathematics) Run-time system Computer programming Subset Heegaard splitting Bit rate Different (Kate Ryan album) Network socket Cuboid Stability theory Social class Physical system Programming language Moment (mathematics) Electronic mailing list Sound effect Regulärer Ausdruck <Textverarbeitung> Complete metric space Arithmetic mean Data management Process (computing) Befehlsprozessor Vector space Telecommunication Right angle Cycle (graph theory) Quicksort Functional (mathematics) Implementation Server (computing) Web browser Distance Code 2 (number) Revision control Latent heat Goodness of fit Term (mathematics) String (computer science) Data structure Metropolitan area network Computer architecture Cellular automaton Expression Line (geometry) Interprozesskommunikation System call Vector potential Word Integrated development environment Personal digital assistant Video game Internet der Dinge Active contour model Window Library (computing)
so uh before start and just before I get into the details of micropipette excuse me little story so and 2006 I fell in love with over 10 years ago with Python and when you fall in love you just wanna spend all your time with your loved 1 and we did we did command line script in web applications graphical applications networking coroutines we spend all our time but then on 1 fateful day something horrible country the so who we hear knows all can't years ever used it heard it raise your hand if you have it good makes even more exciting so kindly is kind of a is itself a very useful and then this is concave that
looks like and what it lets you do is a small after the resources on your computer that a very like the key thing to do every but you know you can see the CPU RAM disk uses of like that and this
is my Conkie set up that I have on my computer so then so I could do
all the stuff of Python very happy with it and then when I encountered Conkie I went
to configure and customize and as he's 300 built-in widgets for CPU rounds all that stuff but then you always have something customers right that's relating to yourself and for me I had these 5 things I want to watch for and Debian packages having updates my Domain Controller passwords so the way Kong words joint standard and there was simple can where you just in you config file outputs and scripts and then you specify how often you want to be called physical so created 5 steps in Python of course the love of my life and death I wanted to update every 2nd so as a good case Conkie is gonna spawn these processes every 2nd and have filed the scripts to say just that a calculation was like 0 crap going run this 400 thousand times every day so that maybe that's OK let's see what happens so I did and initially I just made a very simple Python Hello world and kicked off and said OK what's this going do and cocky itself is implemented in a very efficient fashion because it would be really lame if you ResourceManager load on your system so why did this my CPU usage shot up to 15 % but then instead of Python scripts when additional Hello world bash we use a lot less CPU and so I ended up just not using Python for those uh slapping together some batch scripts and not doing it every 2nd in every 10 seconds or not but you know over time over the years I found stuff that Python hasn't worked out for me and so is frustrated me because I hate programming in batch I mean if statements in for loops and batch are very ugly and painful to work so it frustrates me if I can't use and I and the this who kind of other examples separate the college example and where there's issues sometimes it's a and performance and maybe 1 a measure the performance and 1 example is you new text editor you can hook and an electrical every save and I use them whenever I save my source code it goes and then runs like a goner and then if you just check this on a very like simple file like a single line print hello world and takes over 3 analysis happen now from usability perspective right anything beyond 100 ms people perceive very frustrating when you're editing cold and save and you feel this kind of black interesting in the far but you know it's very useful the habitat beat tracking selecting sites and what i'm gonna kind of presented in this talk is an alternative to see Python that has a much faster start time at which is using much but but was and later another use cases batch completion so Clariant command-line doing tactic that you're typing and you really wanna responsive uh experience in assigning the 400 ms is something what's going on and you know whatever so with that as a goal cool they have about condition for but I set it up and find it cut at out you know sometimes it can take up to a 2nd you know to to kind of do the best completion because it is a little heavy to important execute is very frustrating xi facade using and I thought maybe there's something wrong with my tunnel SSH which may be the networks what not and and so in the end I just stable and start using it because when something gets so slow then it turns usability than impacts functionality so you were in a try and look at 2 things the measure and script execution so we focus on let's say you have a specific script program and you're going executed and starts going on and you want these 2 simple things know what was the elapsed time and also the clock cycles why the fox and it's it's funny I mean like from the end user perspective all they care about is the elapsed time but when I so during these benchmarks on this lot of which is 5 years old and the mother does not which is in your computer the under the DB a lot faster In natural you know and that can be kind of frustrating when you're comparing results what school is if you compare the clock cycles will be the same way is if you have the same or less the same and software generally it's going to go through the same number of instructions that to execute so clock cycles can be useful to to kind more fairly compare an elapsed time is the real kind of experiences and then hold when you're looking at look at the case of calling the maybe you're planning to actually execute some sort of crazy job that every 2nd is gonna do some polling in an and you want to kind of measure you know how heavy is this going to be on the CPU and so clock cycles of
so why measure 1 when you can measure 5 so what i citizens and these benchmarks and I know already that you know bachelor world can be faster than survival and what it is they said let's go into the next uh 5 interpreted languages so we have biphone banish pro the will of all the lines are interpreted like Python so they have variables of functions and for loops and so acidic is a fair comparison is be unfair to compare it to see are going know there's compiled and so I created these 5 files and then essentially they're all the same implementation adding will plus so all of them the execute them in in language they should output to and and then they were gonna run all this performance and see how it is Python compare you know is is unfair you expect like a very fast performance time from an interpreted language you know and and the first one in add up you I will be used for the Python 2 . 5 test as well as Michael Python so the same script will run it through both the interpreters and see the different
support so how do we measure execution time the easiest way is type and this is available and national on Unix systems like that it's my ends now this time is actually built tend batch and is different than a time in a binary you find you know on on the on your computer so just important to know that distinction December command line up the different when you look at the man page for time it's likely this time and the generally it's very easy way of just measuring so you the time but command and the only number you really care about is the 1st number that's real and under real and this shows that you know what I just run a script that principal plus 1 by time takes on this machine and 48 males and now this is more like a thorough approach you know and then this is a very powerful command is very useful so and this is a Linux performance tool that comes you know almost machines and you can you package mentioned stall and the pressure command has a bunch of supplements and this suspect this that supplement and I just highlighted in red some things that you know are worth knowing what is it downtrodden so anything you benchmarking you don't just take 1 sample right because it's not really going to be indicative so what's nice about this in the same command unlike time you can say hey to 10 shots and 10 a sample of 10 and then give me the aggregated results so it's things running for 10 another thing to notice the context switches so uh if you're running eventually benchmark for you trying to measure the performance of a piece of software and you're writing a lot of other programs on the computer it's going to give you an consists results because you know the CPU switching context switching between as other CPs other processes so then when you get your contacts which is it means this is getting you know you ran the benchmark on a machine that uh they have a lot of others then now we can get cycles number so and when you just in Python say print was once you know it goes through 15 million instructions per CPU cycles which is that surprising and I don't know much about my computer architectures he is but that's that's the controversy then the last thing and is the elapsed time and so you can see that uh this took 17 . 8 ms and lived in the percentage assigned opossum minus the variants so you just wanna keep an eye on on this because if it's in a post 1 % for instance fine but like if you have a 100 cent the into 2 % variants it means you're getting a lot of inconsistency between these different samples and you should step back and pole I should trust this number right uh so the
now I jump back to my love affair with Bible and then I bumped into this little beauty earlier this year purchases on January 15 10 years after the 1st discovering by the and it's on a popular Internet of Things hardware and I gonna just to tinker with it it's a microcontroller running a megahertz and amazingly unit can run python through this implementation for pattern which runs in the middle so I got I just got to play the Internet of Things but and then I started playing
around with Python and I discovered that you can install on servers and desktops it's not only from the control is 4 so then I started to compare the performance of like a Python begins regular Python and you'll see the results the just the brief on on like a python it's it's a lean implementation of Python 3 they had to uh be much more cutthroat uh about with implemented because targeted for microcontrollers which have limited CPU and RAM and so what they've done is they've taken a selection of core libraries implemented them and and they had this naming convention of putting the the letter you like microtonal in front of them so you have so those are highlighted the socket is just like the Python Standard Library socket and that lets you open issue the connection so you can do that at the client server to spin UDP you'll spot processes retrieve output and Jason ring expressions times itself it's got a nice mix of quite and useful uh packages from taken from the core library so the 2nd the numbers so
when we run those scripts and just add 2 numbers will plus 1 right this is a single text file with a single line of code as interpreted in number different languages we see Python is like way out there it's you know around 18 15 times slower and then all these other interpreted language Lua batch Pearl what's really interesting is micro Python beats them all and you know and it executes so fast that just adding these 2 numbers right and I was really surprised that you can do this in 100 ms is in and once you get to that level just the process of spawning a process of a text file reading the contents compiling an and then it's amazing that it doesn't and under 1 millisecond self and I really encourage people to explore uh micro Python you know outside my controllers you know even on the desktops and servers and it could be useful in a whole array of so those same sort of this is that I use that same tools that proof for both the set of graphs and this is in terms of the CPU cycles and so we see that you know Python you 46 million CPU cycles but like a python only need 1 and and so it's a much kind of more lean implementation and so now you know and it's it's about like biggest 30 or so times less which means that I could probably uses them for and then I wouldn't get this time CPs so am cool so the
next part of the presentation is just looking at uh a real-life implementation using micro Python selected the and to do something useful so and so what I've done is they've done and get bash completion in using micro Python and what's cool about there is the body has such completion delta so we can compare the 2 in terms of performance and in terms of features and besides the there's also stuff that you can do much Python that would be frustrating to do and other languages that have better sort of have the core of certain that batch if you had to go and connect and to to a web server and fetch a listing of the wheel packages which you'll be doing in the frustrating to do so them go so let's see how the built-in that performs if we just import that because you know when you do condition that it imports and then calls it just importing the package but haven't called auto-completion you know on on on my laptop takes 1 2nd and a lot of the machines of a scene and it's it's heavy and so you really going to feel that you know when you when you press studied the general you know benchmark is you want to be under 100 ms usability generally anything under 50 ms people can perceive you know the cap received more than 24 25 frames and but stuff more than 100 ms if the as you get a feel that kind of jittery life and anything the terminal type the other thing is the built-in that that completion it only completes the succulent did you get that says install whatever but it won't complete the needs of packages and stuff like that so we're going to take it to the next level and and we will also complete so now the the code that I'm showing everything it's more piloted demonstration it's not there the perfect kind of implementation and so for the demonstration have a local pipeline manner and that of creating what I've done I just got a whole bunch of wheels over 400 Wilson just have servants and uh honest adequate to religion sex and death and to do that we just set these environmental variables and so that it looks in that location and binding is local laws but you can never do it on any web server new forms and ends so will hold a complete the supplements and implementation and will also every time you press it too will connect activation be checked you know genetics what's a set of wheels so the moment a new wheels published were put on the web server in press tab will autocomplete which is nice because some times is a trick when not completion slow people catch the results but it's frustrating because often it bought out by stimulus will be like for each call it also can be noticed the packaging in uh the versions as well and so you'll get all that information in the terminal when you to type completion so what's the most useful about this but you know uh there's different ways of doing much completion and usually if you want to run line each time then you have a badge function and then uses complete fashion and I've set the command option names because when we say that stall you know click equals equals 1 . 0 equals is actually a special characters and Bashir needs to be escaped you'll see that so this is how you say you know allow special characters Parliament and then you'll see that all the work is being done by this that comp that look at a source coding years he's 1 argument uh comp seaward income the word if you're completing the 1st an argument that it was the 1 completing 2nd argument and will see 2 and will need to keep track of that because if you do put space that than we wanna say install freeze covered if you do that stole space and time we wanna give the package needs and so so so the cold so these are all the packages that will be using implementation and at the top that just says hey you select a Python interpreter instead of a part and you request is just like very similar to request that well known love and accept it's implemented to work with you sockets more kind of lean implementation and then and then there's a western answers so this function does the bulk of the work and the 1st line will go and connect and using requests to the euro provided you know and fetch vegetation up and the 2nd line will go parceled and get the list the wheel files that are posted on the button and dead then we just looked through the wheels and and passed by action and splits my dash to show the name and version and there no we'll file formants physical because in the file name and its encoded the name and the version so you can get all the information so and then that's it that's the whole function the lower there is the main and all he does is he checks if you gave argument 1 then this is the list of supplements that will complete and if you give argument to then we do want a hard code the or else so will get what the user specified environment variables beginning that links will call get packages In a printout distilling limited so had to do batch completion you just have to spit out you know a list of values that I space-limited New England and this let's compare the performance so we know what the just importing that for a completion can take up to 100 a thousand milliseconds so how does this fit so we see that 1 is just simply speeding up the commands he doesn't 3 . 9 0 seconds which is way under 100 and then when he connects to and that packagings and it has to fetch the package names that he opens HTTP connection gets up causation and prints out all 11 ms which is fantastic performance we more than half of what we need so the libel that is so so the 1st thing that I want
to is just set a clean environment so this is shouldn't have anything stolen and we can start getting so so let's just look at the conference so far I C 1 you should tell me the canons which he does if I say to you should connect to 1 observer and we can see engine experts in here is just has a bunch of will foster just opening this URL and and fetching and uh spitting out the packagings equals the now let's just
check how many packages we have the demo 409 wheels and that was studying some completion so if we do patterns that no well it's stupid Tad so this is working then we can put commands can take that as the options installed now I can see Tab and it shows me 409 packages so you I can go and say OK well the packers between c and you'll see that it's showing the click and color armor you and now and so I can go and say OK I want so click let's say original . 0 detector so you see the equal signs that escaped correctly and let's install color on it we can do the previous 2 seats installed and 1 of its great click to 6 people but the see propres we conceive that's been upgraded to so n so that's it
yeah distance votes yes
and that's the whole presentation here the the and 1st of all thank you much for AutoCAD so she really interesting approach to use for completions but I'm wondering we should agree this this might require and these compatible or incompatible or by phone and like is the only critical chance to warrant a larger existing programs from parts into my so thanks for the question questions so and it is actually challenging like if you
just took a big codebase like
that that for example you know there
were flake you know neutrons importing it that is just a lot of that's not gonna work and by you know what would be really great and an almond thoughts about this you know and the keynote few days ago is a let's say you have the specification of Python the do nice if you could go and then have formalized subset of by for the micro by specification and then people could come in and be like a I'm going to write vacates so that it supports both you know the the full-fledged by phone and my Python and that's happened in the past with Ji found for example and you know the Django for a long time supporter Jonathan and and when you'd run Django it would make sure that this code would work equally well in Python as well I even though the Jive and runtime environment so different so um it I think you right now at this stage there will be issues but um I think there's just a a lot of opportunity and potential and and Michael Python is just is really stable and an so I think there's a lot of potential and uh afford an opportunity for it it the the thank you that was pretty informative I just wanted to up to that the room and apartment the units has long distance for solving this sounds like they would look at the time the the slack in you just on non pulses because they do not want them to start up in start time of quite confident but because this is a little misleading but to say that all the way the performance by goes maybe this is both stopping time there no to books not being used that brings the you what approximately gets the same time because this is all about loading library it's rough the about the like in case of also known as a window manager when you were writing screws they got to want to round off the than maybe we just this bond run boxes and then use just some the something old often like from you would just is not the you know they kind of fire them talking income and then you you would get the something real time because that's what you want to I got a new 2nd not so much we it yeah you know what is 18 ms right like when I 1st saw it was not but I think it's always good to look around at what's that so when you can see an implementation of a subset of Python for you know running at 30 times more maybe the user's opportunities that can arise from so think of the presentation more as an eye-opener 2 possibilities term but what she said is very important is true I mean 1 could argue that the architecture of calling the is messy you use this font process every time right even if it's fast election feel very kind of sloppy maybe I should have a long-running process and induce interprocess communication rate like D-Bus or something like that that 1 of the reasons why yeah calling other processes and just getting the output is such a popular method of cooking is because it's so easy to implement and it's very widespread right what Kant uses it then uses that he makes is like when then musings and that's the you know when I press save each time it's conflict and you know even though maybe some of that made a flake server and then it could kind of call more efficiently right and the same thing is true of a lot of uh Jenkins as an example you know like when it wants to look into Python a lot of times it's a spontaneous process so it's kind of in a very popular way of cooking in uh executing programs and is capturing the output but you know uh as he said it's not so affair you know assessment of Python I wouldn't put as a criticism of the of Python because I've never had a decent system that greater production environments this sort of issue for of i have a sort of issues I create a long-running processes and then do some and process communication but if the Ch of Partington presentation come on the market president of mortgages shows was when I was a split between 1 brought into and through your use of the 2 words 23 yeah good question like a Python is and that also when they created they said 1 I can support Python 2 so the whole thing is Python 3 from the start and so all yet so that's has a very important thing mentioned because it is you don't go back at the station and so it's all completely clean implemented in Python 3 and it's the same exact syntax in this some differences some edge cases but you know you have all the same data structures functions classes in the rest of it the more I work with that and the more I didn't encounter any surprises at the it thank you for your talk about the question do you seems to me like a vitamin I will have an effect this in fact only by means means that's a say lasted again it just kind of thing my bite the good things in life by 11 and this thank implementation in uh well I hope so because you know it before and like before January 17 someone came to me and say hey Python to start in 1 millisecond that buzz off you'd like of seconds is great if such a powerful and trade like in a way you could argue about you know or or or some other languages a very limited you know it in a that powerful is by but I think now that Michael Python has been implemented and there's such a growth in Internet of Things and so much contribution like by for the stability I think the make as the question again you know that Norman again raised about like you know pipeline does it have to follow this C Python thing you know maybe we can have the specification maybe we can have a license the station and then a subset of you know or like is that maybe some of these strict selector Python has a c by the could use but not sure about that because you keep in mind when Python those drawbacks to there's no multithreading process and that's how I was able to get away with this kind of all over oversimplified uh but but it's it's it's amazing you get that much functionality with that few CPU cycles unsurprisingly Ch hi thank you for the great talk them and we understand them problems with legal but with the gold it is the same and you see that little or these the like son of library but what kind of opportunities the contour so good question for me why implemented this uh so 1 get examples requests requests is such a popular library and that evidence like I do I have to directly talk to soccer because they give you use often but they don't give you um the Python 3 libraries you have for the HTTP client and so imagine bike riding low-level socket code vector each of so that's a pain and so on you know at the micropipeline guys made you request and and they made you jot Ji Sun in regular expressions it's I think a lot of the popular libraries are there the the and popular functionality but and it's it's not that this survey that the Python code that you're right but the list is different for the string is different the functions through the coding is cell that's exactly like regular Python there which is this is just a you know if you take any of these packages and you try important in all collections you note and and or even or even as rate expression you know you don't have the full RE and so you know uh they've made you are in which it has a lot of the functionality but it's limited itself you have to change a lot of you import passing you have to be thoughtful which I think I think if you know the code isn't so crazy and complicated man then it's definitely worth it because but he saw that the purported completion it's not like 10 thousand lines of code and you know and it I think is the 15 or 20 lines of code so you if you have this kind of small thing if you make something new I think it's definitely worth of alright me during moment was reported in the if