Thoughts on the Future of Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 115 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/58797 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 202146 / 115
1
3
19
25
31
34
36
38
41
42
44
46
48
52
60
61
62
65
69
73
76
82
83
84
92
94
96
103
110
113
00:00
TrailNumberSoftware testingWeb browserPoint cloudShape (magazine)Enterprise architectureCodeGoogolFormal languageJava appletType theoryCore dumpPoint (geometry)BitExpressionOrder (biology)Social classCartesian coordinate systemData storage deviceScripting languageEmailElectronic mailing listProjective planeIntegrated development environmentMeta elementData managementObject-relational mappingSoftware developerContext awarenessRevision controlLocal ringInsertion lossBuildingComplex (psychology)Level (video gaming)Statement (computer science)CoroutinePhysical systemPattern languageMultiplication signTask (computing)Strategy gameMathematicsGame controllerDecision theoryEvent horizonCycle (graph theory)MereologyThread (computing)Open setGoodness of fitInterpreter (computing)Loop (music)Covering spaceWeb pageArithmetic meanRow (database)Operator (mathematics)Flow separationDifferent (Kate Ryan album)ResultantIntegerMatching (graph theory)AreaString (computer science)Web 2.0Parameter (computer programming)Computing platformParsingVariable (mathematics)MultiplicationVirtual machineGene clusterInformationProcess (computing)Repository (publishing)WebsiteRight angle10 (number)CASE <Informatik>Library (computing)Presentation of a groupCurvatureSoftwareSlide ruleOnline helpNeuroinformatikMoment (mathematics)Shared memoryGroup actionSoftware frameworkDirection (geometry)Term (mathematics)CloningContent management systemSpacetimeContent (media)Device driverFocus (optics)User interfaceWordObject (grammar)Internet service providerInformation technology consultingMobile appBackupBasis <Mathematik>EvoluteBoilerplate (text)Ideal (ethics)Extension (kinesiology)Pointer (computer programming)Error messagePlastikkarteLine (geometry)Limit (category theory)Expert systemDirectory serviceOpen sourceTypprüfungTrojanisches Pferd <Informatik>Complete metric spaceFile formatWritingMusical ensembleParallel portSpeicherbereinigungException handlingDebuggerPerspective (visual)Information securityRepresentational state transferFunctional (mathematics)Vector potentialMeeting/Interview
00:45
SoftwareInformation technology consultingCore dumpSoftware developerLevel (video gaming)Pauli exclusion principleProcess (computing)Open setKeyboard shortcutData typeBulletin board systemCoroutineFluid staticsMathematical analysisThread (computing)Variable (mathematics)Similarity (geometry)CodeOperator (mathematics)String (computer science)Statement (computer science)ParsingPattern languageParameter (computer programming)Spherical capRegulärer Ausdruck <Textverarbeitung>Self-organizationMathematicsRead-only memoryKolmogorov complexityTraffic reportingInterpreter (computing)Type theoryCycle (graph theory)BitMathematicsMoment (mathematics)Electronic mailing listNumberTerm (mathematics)Line (geometry)Row (database)Right angleOperator (mathematics)Different (Kate Ryan album)ParsingInformationMultiplication signCodeOpen setRepository (publishing)File formatPeg solitaireComplete metric spaceContext awarenessException handlingData storage devicePattern matchingEmailInformation technology consultingSlide ruleSoftwareData structureObject (grammar)Software developerMereologyOrder (biology)Arithmetic meanMeta elementTask (computing)Game controllerSocial classNetwork socketEvent horizonLoop (music)Local ringWeb 2.0Formal languagePhysical systemComplex (psychology)Matching (graph theory)Flow separationInterpreter (computing)Statement (computer science)Meeting/InterviewComputer animation
08:55
Kolmogorov complexityReduction of orderInterpreter (computing)Adaptive behaviorRead-only memoryInformationData managementProcess (computing)Front and back endsPlastikkarteImplementationServer (computing)Mathematical analysisNormed vector spaceStandard deviationEndliche ModelltheorieCore dumpSoftware developerWeb 2.0Formal languageShared memorySoftware frameworkRepresentational state transferEnterprise architectureFront and back endsError messageGene clusterLimit (category theory)MultiplicationThread (computing)NeuroinformatikProcess (computing)Software developerBasis <Mathematik>Library (computing)Flow separationDirection (geometry)Term (mathematics)Information technology consultingElectronic mailing listCanadian Mathematical SocietyWeb browserProjective planeContent (media)Integrated development environmentDevice driverMoment (mathematics)Cartesian coordinate systemData managementComplex (psychology)CodePointer (computer programming)BitPhysical systemEvoluteNumeral (linguistics)Task (computing)Interpreter (computing)Diagram
16:54
Enterprise architectureComputing platformDataflowScripting languageProcess (computing)SoftwareSoftware testingRevision controlPatch (Unix)Core dumpAdditionSubject indexingKeyboard shortcutStandard deviationProcess modelingMathematical analysisKolmogorov complexityIndependence (probability theory)BuildingPoint cloudWeb browserStack (abstract data type)Open setLambda calculusGoogolAsynchronous Transfer ModeShape (magazine)Java appletInformation technology consultingInternet service providerSystem programmingTrojanisches Pferd <Informatik>Greatest elementVector potentialError correction modelEnterprise resource planningFocus (optics)Core dumpTypprüfungFlow separationType theoryBitTrojanisches Pferd <Informatik>Multiplication signSoftware developerElectronic mailing listAreaEnterprise architectureJava appletFunctional (mathematics)Complete metric spaceGoodness of fitCartesian coordinate systemWeb pageWeb 2.0WritingFormal languagePlastikkarteProjective planeInformation securityVector potentialStrategy gameDecision theoryCycle (graph theory)Covering spaceCodeMoment (mathematics)Web browserScripting languagePoint cloudCASE <Informatik>MereologyArithmetic meanResultantComputing platformComputer animation
24:52
Mobile WebParallel portAdditionGraph (mathematics)Data structureMiniDiscRevision controlInformation securityCodeLevel (video gaming)DisintegrationIntegrated development environmentBulletin board systemFront and back endsDatabaseEnterprise architectureSoftware developerOpen sourceMobile appParallel portCartesian coordinate systemSlide ruleIntegrated development environmentState transition systemFocus (optics)BitExtension (kinesiology)Boilerplate (text)AreaLibrary (computing)Internet service providerLevel (video gaming)Goodness of fitFormal languageInformation securityRevision controlPoint (geometry)Direction (geometry)Computer animation
28:06
Computer animationLecture/Conference
28:34
BitWeb browser
Transcript: English(auto-generated)
00:06
And we're back on the conference track. Those of you who have been here this morning have missed the cancelled Zach Hatfield's dot talk, Stop Writing Tests. We have managed to find a new place for that and it is starting now in conference number six in Silly Walks.
00:27
And that talk will also be available as a recording so that you don't have to watch it right now. But if you were waiting for that, that's where you can find it. And talking about Silly Walks, in the spirit of Monty Python, I can now say for those who have just missed Marc-André Lemboek, here he is again.
00:46
Hi, thank you for the introduction, Martin. Yeah, and we're short on time, so I will let you take over right now. Okay, thanks. Bye. Okay, so I'm going to talk about thoughts on the future of Python. These are my personal thoughts,
01:05
of course. So there's nothing official behind this, but I'm going to go through a couple of things that I think where Python is, where Python could be in the future, and hope you find this interesting. The talk is actually, it was written for 45 minutes, so I have to compress a
01:23
bit. I won't go through all the details on the slides. I will post the slides after the talk. So a bit about myself, I'm Marc-André Lemboek. I've been using Python since 1994, so quite a long time. It also tells a bit about my age. I studied mathematics. I have a
01:41
consulting company. We're doing Python projects, and I'm working as a consulting CTO and CR software architect. I'm also the EuroPython Society Chair at the moment. I'm the founding fellow of the PSF, Python Core Developer, and I am based in Dusseldorf. So what's on the agenda
02:02
for this talk? First, I'm going to go through the Python development. Then I'm going to have a look at the growth in the market, perspectives for the future. I'll give you a few ideas about my wish list for Python, and then perhaps we have some time for discussion. It's more probably that we're going to have those in the breakout room afterwards. So let's head on into the
02:25
overview of the Python development. So setting the stage. Python has been around for a long, long time. Is it still evolving? So let's have a look at the history of Python. It all started in December 1989 when Guido van Rossum started to write Python. He was working
02:44
on the ABC team at that time, and he thought, well, I would like to do things a bit differently. So he started writing Python. In the 1990s, basically everything was done on Usenet. So he posted things there. There was a lot of development. It was a very, very small community
03:05
at that time. Over that decade, basically Python grew. Then in 2000, the zero years, what happened is that the development opened. So we had more developers come in. We had people
03:25
who got access to the repository. The Python Software Foundation was initiated to own the licensing and to own the IP rights. And then in the tens, what happened is we moved to a different repository system. So first we started with CFS, then we had Subversion,
03:45
then we went to Mercurial, and then we went to Git eventually, and we started the Python two to three transition, which is over by now, luckily. So more recent developments, I'm just highlighting a few things here which are very important. I think the most
04:03
important development are type annotations. So what we have now in Python is you have the possibility to say, okay, this variable is going to be an integer, this variable is going to be a string, or there's a way to also define more complex types in Python.
04:22
You can use those type annotations in several different tools. You can have type checks in your IDEs, for example. You can use them in data classes to define the variables, and then data classes make it possible by using the type information to more efficiently store the information. The other big development is AsyncIO. So AsyncIO is about
04:48
basically putting an event loop into Python and then managing that event loop in Python to have parts of your code run as coroutines, meaning that they run for a certain time, then they give up
05:05
the task. They say, okay, I'm going to have to wait on something. They give the control back to the event loop. The event loop then takes the next coroutine, runs that, until that again says, okay, I have to wait, so please go to the next one. And that makes it more efficient to run Python code where you have lots of waiting going on, let's say, for example, web socket code.
05:26
Other developments are context bars, which are local variables, but for Async code, so that's like it goes along with the AsyncIO. A very important development was to have the built-in dict be an ordered dict, so it preserves the insertion order. That made a lot of things that
05:46
we had to do using meta classes, for example, in ORM classes, made those go away, so basically they got a lot simpler. They could remove the meta classes and now we have access to the order
06:00
in which attributes, for example, were added to a class dict directly in there. And data classes already mentioned it's an efficient way to store records. We also have a number of new operators. I'm just picking out a few, so we have a new dict merge operator. We have the well-rest
06:21
operator because it looks a bit like a well-rest. It's basically an assignment inside expressions, which makes a couple of things a bit easier. There were long discussions about this. I won't go into detail, but you can read it up on the mailing list if you like. A very nice development was f-strings, so you can format things directly in, well, you can do
06:44
templating inside Python with the direct access to locals. These are not only nice to write, they're also efficient because they execute faster, and this was a vastly successful thing. We have positional keyword only parameters. These were basically backported from the C API,
07:03
which had them for a longer time already, and now they're available in Python as well. There's a new parser called PEG. This allows us to add new keywords to the language without disrupting the complete code base because they are context local. Then in 3.10, we're now getting the switch and match statement. This is for structural
07:24
pattern matching. You can also match on type and objects. This is something that's new, and it still has to be seen how well that's going to be picked up. We also had some organizational changes in Python. We have a new release cycle. We have
07:41
yearly releases now. We have a steering council that took over after Guido stepped down as the BDFL and retired. Then after a while, Guido got tired of being retired. Basically, he started working again, and he's now working at a team at Microsoft with the idea to improve
08:04
the performance of C Python, which is very nice. Very recently, we also have the first PSF developer in residence. The PSF now actually has a higher developer helping us to reduce the number of open PRs. What's currently under discussion? Like I said,
08:24
there is a strong discussion about improving C Python performance. This is not new. This has been ongoing for years and years, but now we have a dedicated team to work on this. They are working on a specialized adaptive interpreter and also adding many small
08:40
improvements. You will see some improvements in terms of performance over the next few releases and years. We will have enhanced error reporting. Instead of just reporting on which line something went wrong, Python will also be able to report on which column or in which column something went wrong, which is going to improve the error reporting, especially for
09:03
newbies, and also in some cases, of course, for experts, because it's not always obvious where an error occurred if you have longer lines. There's a development to add sub-interpreters to Python. The idea there is that to work around the Guido limitations, the global interpreter
09:22
lock limitations in Python, which makes it hard to run multiple threads with Python. The idea with sub-interpreters is that you can have one sub-interpreter per thread and then have those run on a per thread basis. We are also working on the C API. The C API is being
09:47
basically, let's say, it's being resorted, so more of the APIs that we have are moved to internal, so they're not going to be published anymore to extensions. The complexity is supposed to be
10:02
reduced. There are a couple of projects on the way to help with all of this, to also maybe change the garbage collector, and to introduce something called the Python handle instead of using pointers. Lots of work is being done to improve Python. I think in terms of the original
10:21
question, is there still development going on? Is evolution still happening in Python? Definitely, the answer is yes. There's lots and lots of stuff coming. It looks quite good on the Python development side. Now, let's have a look at the Python growth in the market. If you look at
10:40
the history, everything started in 1989, and then very slowly, things got released to the news groups. The academic world started picking up Python a lot. We had the first workshops here. We had the first company, eSharp, use Python for e-commerce. We even had a browser implemented
11:02
in Python, which was something. It was a pet project of Guido at the time. You can see down here, Numeric was started. People in science were trying to use Python for their tasks. What happened in the next couple of years is that Python started very much embracing the web.
11:24
The first development was the Zope environment to publish to the web. That was very much object-oriented. Zope was considered to be a bit too complex, especially for users. What happened is that Plone was on the rise. Plone is a CMS system. You can use it
11:47
basically also for content management and knowledge management in the enterprise. In the end, Zope, Plone was built on Zope. The complexity in the underlying code was still
12:01
there. If you wanted to extend things, you still had to go into Zope and do stuff. That was considered not to be a good thing, especially for new developers. Django entered the stage, and Django made things easier. Down here on the axis, you see what happened in the community. There was a Python consortium
12:24
built and created to support Python. The PSF was founded in 2001. EuroPython started. PyCon started in the US, and things were going well in terms of community building.
12:40
After a while, Python started entering the mainstream. More and more people were using Python. When you went to customers and you told them about Python, people actually knew what Python was, which was exciting. If you went as a consultant to a company and told them that you were using Python, then they'd say, oh, yes, great. I know that
13:02
language. You didn't have to tell them, okay, this is a new thing. It's going to be a bit risky for you, but we can still try it. I'm going to tell you it's definitely going to be worth it. That was the discussion that we had in the early 2000s. You can see down here that in the science area, lots of development started to happen. This is actually the next big thing
13:25
that came after Django. You have a huge list of libraries and projects pop up in the data science space. This is where we are right now. If you look at the development of Python
13:40
and how Python is being used in the industry, nowadays it's mostly being used in data science. Of course, we still have a use in web. That's not going away anytime soon, but data science is the big driver that we have at the moment. Now, the question is, what's going to happen in the future? This is basically what I'm going
14:04
to try to make a few suggestions for in this talk. Let's see where we are. Like I said, Python is still big in web, but web works differently now than it worked a decade or two ago. Web is definitely no longer just about publishing content to the websites. Nowadays, it's mostly
14:27
about providing an REST API ideally to provide data to the front-end. Then the front-end is typically developed in JavaScript using one of the many frameworks that you have in JavaScript.
14:42
We also see asyncio being used a lot in the web to make things run faster. All of this is going in the right direction. Python is indeed very well positioned in the web, not only because of Django, but also because of or like the CMS or Django CMS. But
15:01
it's also well positioned in terms of API backends. We now have fast API, which works based on the asyncio framework. We have Django and Flask REST frameworks, which work synchronously, but now they also are getting async support. We have libraries that help you with GraphQL.
15:22
You have Graphene, Ariadne, Strawberry, TottyFlat, and several others that help you push data to the front-end. I think the story for the web is pretty good. It is already very big in data
15:40
science. Python and R are basically the two main languages that you can use in data science. This has really dramatically increased the scope and the market share of Python overall. What I'm seeing now in the market is that, like we had in the keynote, the heat framework
16:00
nowadays, everything is about distributed computing. We are moving away from having just one application sit there, maybe use multiple threads to run something. We're replacing that idea with moving to multiple processes or even multiple machines, clusters working together to solve a problem. That's where everything is moving.
16:24
The Python GIL discussion that we had in the last couple of decades is actually less important at this point, because the GIL only runs in one of the processes. If you have multiple processes running already and you have clusters running multiple machines, then the GIL is
16:42
not so much of a blocker anymore. We have two markets covered. What's the idea in the enterprise? This is what I'm seeing as CTO of companies. I'm seeing that Python is entering the
17:00
enterprise via data science. It's no longer the case that Python is just being used for implementing websites. It's also that Python is actually powering very important parts of enterprises nowadays, because everything is moving towards being data-driven. Data-driven means that you take the data that you have, you try to analyze it as fast as possible,
17:22
and then you try to build your decisions, the business decisions, on the results that you get from analyzing the data that you have. Python has a very strong footing in that area. We have, for example, Apache Airflow, which is being used for workflow. You have Superset being
17:44
used for business intelligence, so dashboards, graphics, and stuff to put directly on the web to run fast analytics. You have Apache Spark, which is a data platform which is distributed, and it runs PySpark scripts. We are actually well positioned in that market as well,
18:04
and this is going to grow. The question is, is Python really ready for the enterprise? My take on this is not quite there yet. We still have to do some work on that. This has to do with a couple of things. Now, putting on the CTO ad here,
18:29
you're responsible for developments that you're doing in a company, and so whatever decisions you make, you have to tell your shareholders, you have to tell your CEO, this is the right
18:41
thing to do. There are a couple of things that are a bit worrying when you look at the Python development from the CTO side. One is the yearly release cycle, for example. You get one and a half years support for every release now with a new cycle. You do get faster
19:04
access to new features, but what's going to happen in companies is that they will probably not switch to new releases for every single release. They will skip releases, and so because you only get one and a half years support for each release, what will happen and what we
19:20
will have to see, and we will have to have companies in the Python ecosystem providing this, we will need commercial support for Python. After those one and a half years, you get support from the core development team. You can then switch to commercial support that you get from companies. It's not, maybe I put it a bit wrongly, it's not that
19:43
bad a thing to happen, but you need to find a different strategy to basically back up your decision to use Python for this. Now, how's the core development helping in this? Core developments are good. We've added type safety, for example, type annotations. This is
20:03
something that's very much liked in the enterprise. People have always had an issue with not having type safety in Python, so this got added now, and you can write code using type annotations, and in fact, in the enterprise, this will become more and more of a standard,
20:23
so you will see type annotations all over the place. The developments happening in core development are actually a good thing, and adding new features is a good thing in Python. It's not so good for newbies because Python has grown very fat over the years. For example,
20:42
I just looked at a book in the last couple of weeks to just cover, basically try to cover everything that's in Python. It has over 700 pages, so that's where we are at the moment. When I looked at the tutorial, when I started to learn Python, I don't remember exactly how
21:00
pages it had, but it was definitely a lot less, so probably 50 or 100 pages. This is just core development. Core development is not the only contributor to what Python is. We also have a huge data science community. We have a huge web community. We have a DevOps
21:21
community. For example, Ansible or SaltStack are implemented in Python. We have several other communities in niche markets. Python is growing all over the place, and we have many, many people working on Python's success. It's not only the core development team that
21:41
provides new features. It's also all these other communities. What's happening is that lots and lots of industry leaders are using Python nowadays, and so you can reference those. We have companies like Airbnb, Netflix, Google, Microsoft. All of
22:02
these have written applications, and they have open sourced their solutions and given them back to the community. That's great. Could Python be seen as the cloud language? Well, I would say yes, but only to some extent. There's lots of competition out there,
22:23
so especially Node.js is one of the big competitors in that market. We do have lots of APIs to all kinds of cloud libraries, to cloud solutions. You can write serverless code in Python, like AWS Lambda or Google Cloud Functions, other functions. You can
22:40
write all of those in Python. It runs in the browser. OpenStack, for example, is a complete cloud solution which is written in Python. We're in good shape, and we have market awareness. Could Python actually be the new enterprise language? Well, of course, there's strong
23:04
competition from Java and C++ and C-sharp, to some extent, depending on where you go. The problem there is you have to say nobody ever got fired for choosing IBM, or nowadays, it's more like nobody ever got fired for choosing Java. That's what's happening. If you see the
23:23
enterprise software development market, then still most of the things are being developed in Java. You can also see that if you go to, for example, the Apache project directory, most of these big applications that they have are actually written in Java. Only very few are
23:42
written in Python. Now, how can we enter this market? It's difficult, I can tell you. People love typed languages. They love the security that they get from typed languages. Now, we have type annotations, so things are getting better. I think that we still have to use this
24:01
Trojan Horse card to get into the enterprise. There's lots of potential that we have. These are just a few acronyms that you may or may not know. I put the wording behind these. If you look at the blue entries in here, we only have very few areas where we actually
24:25
already have a solution written in Python. There are lots of areas in here where we don't have anything. This is a huge market. It's billions of dollars in market size. I have a separate talk on that. You can look it up. I held that in 2018.
24:45
I think we will get there. It just takes a bit of time and effort and, of course, people to use these ideas and then go ahead with them. This is my wish list for Python. I have two slides on this, one at the language level and one at the application level. The first one,
25:03
again, from the CTO perspective, what I would very much like is a Python LTS version, so something where we can say, okay, we will have this version out. It's going to be supported for five years and then perhaps you can even get another five-year security fix it.
25:22
That would be a great thing. That would be a big selling point in the enterprise for Python. The other things are not really that major. Of course, everyone wants to have a faster Python. We would like to have mobile app support, but that's coming. People are working on that. It would be excellent to have a syntax for
25:43
parallel execution. Now that we have async IO already for the asynchronous execution, we're still missing a bit on the parallel execution. There are some tools which help you with this and try to automatically run things in parallel, but it would be nice to have a syntax for this directly in Python. We don't have that yet. We still have to
26:02
rely on multiprocessing, MPI, DAS, these things. It would be nice to have something directly in Python. Of course, Python without a gill would be fantastic. This is going to take another few years to achieve. Because we don't want another two to three transition, it's going to be hard because Python without a gill is actually going to break a lot of extensions.
26:25
We have to have a good path towards that. Now, on the application side of things, I think what we need is we need better rapid development solutions. We have good IDEs for Python, but those are not really rapid development solutions.
26:41
You still have to do a lot of boilerplate development regardless of which area you choose. The tooling that we have there is not all that ideal. We need more polished libraries. We need business graphics. We need business UIs, dashboards. We need these things to be
27:00
backed by companies and not only open source. Only is not the right word, but not just backed by open source communities. Those open source communities should have companies within them to provide things like support, backup, maybe extended solutions.
27:25
We need more paid offerings. We need more startups that enter the market and provide easy access to consultants, for example, to provide SLAs, which is something the enterprise always needs. We need something that makes it easier to develop applications in Python.
27:43
Anaconda is already going in the right direction for the data science part. They have very well integrated tools. They have a nice IDE. They have deployment tools. They have an app store. That's fantastic, but it's too much focus on data science.
28:00
We need that in other areas as well, especially if we want to enter all these different acronyms that I showed on the previous slide. Where are we right now? Python has a bright future ahead. Things are only going to get better, and it's going to be exciting, and it's going to be fantastic. This is the end of my talk. Thank you, Fred.
28:27
Okay. Yeah, thanks a lot for the talk. We've had some activity in the chat, but there was not really questions about this. I think you managed to shut them all up. People have been discussing, for example, Python in the browser and mentioned Python. I think if you can
28:46
continue this in the breakout room, if there is any more direct questions to you. And so I'll just take the opportunity to thank you again for presenting the future of Python. Thanks. Okay, bye-bye. I'll hop over to the breakout room now,
29:02
and then we can maybe discuss a bit there. Thanks. Bye.