How OpenStack makes Python better (and vice-versa)
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 104 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21231 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2016104 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
BitComputer programmingGoodness of fitMultiplication signFlow separationPoint (geometry)Web pageMeeting/Interview
01:07
TwitterFreewareSoftwareProjective planeVotingOpen sourcePerspective (visual)Medical imagingSelf-organizationSet (mathematics)Software developerWhiteboardData centerBitInformation securityPoint cloudType theoryConnected spaceProgrammer (hardware)ComputerGame controllerXMLUMLLecture/ConferenceMeeting/Interview
03:38
FreewareConfiguration spacePlastikkarteElectronic mailing listProjective planeSelf-organizationFormal languageOpen sourcePerspective (visual)Computer configurationWebsiteSoftware developerPosition operatorBitArithmetic meanKey (cryptography)Goodness of fitVideo gameCellular automatonGroup actionMereologySoftware testingIntegerInternetworkingLatent heatPoint (geometry)10 (number)Power (physics)2 (number)Computer hardwareField (computer science)CodeRepository (publishing)Open setMathematicsEmailLecture/Conference
08:15
Formal languageSource codeOperator (mathematics)Library (computing)Distribution (mathematics)File formatProjective planeOpen sourceInclusion mapPatch (Unix)Process (computing)BitINTEGRALCodeGroup actionRun time (program lifecycle phase)SubgroupStapeldateiService (economics)Goodness of fitSet (mathematics)Physical systemQuicksortDifferent (Kate Ryan album)LogicSoftware developerOrder (biology)Canonical ensembleSubject indexingMereologyFeedbackSoftwareJava appletDirection (geometry)Point cloudLattice (order)Lecture/ConferenceMeeting/Interview
12:16
Process (computing)Multiplication signElectronic mailing listCombinational logicBitLine (geometry)InformationGreatest elementTwitterExpert systemVirtualizationPhysical systemData storage deviceQuicksortPoint cloudProjective planeAreaKey (cryptography)Connectivity (graph theory)Open sourceSoftwareStorage area networkLatent heatComputer animation
14:43
AverageOpen sourceProjective planeComputer programmingExpected valueStandard deviationRight angleProcess (computing)Mobile WebFitness functionWave packetJava appletEnterprise architectureService (economics)Software developerProduct (business)Multiplication signGoodness of fitCartesian coordinate systemClosed setUltraviolet photoelectron spectroscopyMereologyLecture/ConferenceMeeting/Interview
17:20
Library (computing)Observational studySoftwareProcess (computing)Software testingNumberCASE <Informatik>Classical physicsControl flowDeadlockCondition numberMultiplication signCache (computing)Software bugTraffic reportingLinear regressionException handling2 (number)JSONXMLMeeting/Interview
19:49
FreewareDifferent (Kate Ryan album)Electronic mailing listCodeLinear regressionMilitary baseSoftware testingLibrary (computing)Validity (statistics)MereologyCASE <Informatik>Software bugOnline helpAddress spaceLecture/Conference
20:49
Different (Kate Ryan album)Wrapper (data mining)Declarative programmingSoftware frameworkAdditionLibrary (computing)Goodness of fitSoftware testingCommitment schemeWordJSONXML
21:20
Programming languageSoftware bugPattern languageTraffic reportingSoftware frameworkLibrary (computing)Multiplication signBitProgrammer (hardware)Software testingCodeArchaeological field surveyInformation securityNumberSoftware maintenanceType theoryConnectivity (graph theory)MereologyMathematicsPhysical systemQuicksortLinear regressionRange (statistics)Game controllerHookingServer (computing)Process (computing)Computer fileTemplate (C++)Projective planeCASE <Informatik>Greatest element
24:45
CodeLevel (video gaming)Revision controlProjective planeElectronic mailing listProcess (computing)InternetworkingBitLibrary (computing)Cache (computing)Connectivity (graph theory)Perspective (visual)Computer-assisted translationMereologyPatch (Unix)Information securityMessage passingFeedbackSoftware bugTerm (mathematics)Wrapper (data mining)Extension (kinesiology)Linear regressionSoftware frameworkNumberNatural numberArithmetic progressionQuicksortMultiplication signSlide rulePoint (geometry)AuthorizationSurfaceMusical ensembleControl flowWordSoftware developerRight angleSystem callComputer animation
29:29
Category of beingOpen sourceProjective planeEvent horizonBitLatent heatPatch (Unix)Scaling (geometry)Frame problemPersonal digital assistantLattice (order)MereologyLocal ringComputer programming
31:08
Self-organizationSlide ruleMultiplication signCASE <Informatik>Matching (graph theory)Lecture/ConferenceMeeting/Interview
31:46
FreewareFormal languageProjective planeArithmetic meanPoint (geometry)QuicksortTerm (mathematics)Message passingFamilySoftware developerCodeLibrary (computing)Limit (category theory)Pattern languageCASE <Informatik>Software testingJava appletNumberForestSoftwareRevision controlPhase transitionComputer programmingContinuous integrationLevel (video gaming)Function (mathematics)State of matterMultiplication signOrder (biology)Different (Kate Ryan album)Cartesian coordinate systemMathematicsLine (geometry)Data storage deviceSlide ruleUnit testingProcess (computing)BitMereologyDirected graphArithmetic progressionSoftware bugComputer hardwareStructural loadErlang distributionCore dumpLinear regressionLecture/ConferenceMeeting/Interview
38:21
Queue (abstract data type)CASE <Informatik>Latent heatFormal languageErlang distributionDirection (geometry)Scripting languageVector potentialInformation securityProjective planeCartesian coordinate systemJava appletReduction of orderWritingRewritingRight angleFront and back endsMeeting/Interview
Transcript: English(auto-generated)
00:00
Okay, good morning, welcome, thank you for coming. Please receive to engineer Terry Karas and Doug Hellman. Thank you.
00:23
Good morning, so we're here to talk a little bit about two overlapping Python communities, the OpenStack community and the broader Python community. My name is Doug Hellman. Yep.
00:45
And I came to OpenStack from the Python community, so I've been doing Python programming since the mid 90s at some point. I'm working now for Red Hat, specifically on OpenStack. I'm on the OpenStack technical committee,
01:00
which is the governing body for the contributors. And I'm a fellow for the Python Software Foundation as well. Oh, it's working. Oh, okay, sorry. It's working. And my name is Terry Karas. I work for the OpenStack Foundation. I'm the chair of the OpenStack technical committee. I'm also a Python Software Foundation fellow,
01:22
and you can find me on Twitter and Freenode. And so this, I come from the, from OpenStack perspective going to Python community, and Doug is taking the other way around, coming from the Python community and reaching out to OpenStack.
01:40
And I want to start with a short intro on what OpenStack is. Who knows what OpenStack is? Lucky. I don't know what OpenStack is. So for the few of you that don't know what OpenStack is, it's open source infrastructure software that you can deploy on top of a secure data center
02:03
to make it programmable infrastructure. So it's usually deployed by decent sized companies that happen to own a data center and want to provide resources for their internal development efforts or other types of companies that want to run a public cloud basically
02:22
and offer resources for rent to the wider public. It's not exactly something you would deploy in your garage on a Sunday. And I think that's where the corporate image comes from because the users are large organizations and not really small things
02:41
that you would deploy as an individual. And this translated into the negative corporate image for the project itself, which is a bit unfair because as we'll see, it's actually one of the most community-led project
03:01
out there. In particular, we have a fairly independent technical governance. We elect our project technical leads. We also elect our oversight board. The technical committee is the technical oversight board for the OpenStack project. And the election is based on one contributor
03:22
equal one vote. It's not one contribution. It's not one company. It's really if you contribute to OpenStack, you get one vote to elect the leadership of the project that will decide where the project goes and if something gets done or not done. We also have a set of principles
03:40
which we call the four Opens. The first one is pretty obvious, open source. Everyone does it those days. But the others are slightly less obvious. A lot of people do open source. A lot of people do also open development. By open development, we mean that everything that's happening in OpenStack development is done publicly in the open
04:02
on publicly accessible websites. You don't have to sign up. You don't have to be an existing contributor to see how the development is done. We do all our code reviews, all our specs, all our RC channels, all our mailing lists are fully public and open. And we also run only open source tools
04:22
in our project infrastructure. We don't rely on GitHub. We have even the configuration for this project infrastructure is also open source and committed in our repositories, in our Git repositories. So everything is transparent. We also do open design.
04:42
The design of OpenStack is not done by a small group in a corner and then the rest of the community is there to just do extra work. Everything from base design, new project is done within the community. There is no secret group that does things in a corner.
05:03
And finally, we also do open community. By that, we mean that anyone can race to leadership positions in OpenStack. Especially, you don't have to be a sponsor of the OpenStack Foundation or coming from a large company that's happened to be there at the beginning
05:21
to be an important member of our community. So you can get elected to project technical leadership positions. You can also participate to the project as equals without any attention paid to who actually pays for your work or where you're coming from.
05:42
And we're pretty adamant that we need to have a legal playing field. Means that anyone coming in the project should have equal chances to contribute to the project. So we reject projects that would benefit one company or where contributors coming from one organization
06:02
would have a specific advantage. For example, if you work on, if a vendor wants to propose a project in OpenStack that requires a specific hardware like Cisco Switch that costs tens of thousands of dollars, then that project could not be an OpenStack project because basically people that have access to that hardware
06:23
have an advantage against someone that comes from nowhere. We don't have appointed positions. We don't have dictator for life. We believe that change in leadership are good and that's where that open community thing comes from. And finally, to close this introduction,
06:42
I wanted to raise a bit about the Big Ten thing. We used to define OpenStack technically and so when a new sub-project would be proposed, we would try to assess if that fits our mental picture of what we think OpenStack is. And that was pretty useless exercise.
07:01
We ended up having those discussions. Does this project, is it OpenStack? Is it something that uses OpenStack? Is it something? And that was not really useful. And at that point we realized that what binds us all together is those principles, the way we do development,
07:20
the fact that we are a single community, more than a very defined scope on what exactly OpenStack is and where it stops. And so we moved from defining if a project is OpenStack to judging the community that was producing it. Are they following our principles? Are they coming from, are they part of the OpenStack community?
07:41
And if they are part of the OpenStack community, then they are an OpenStack project. There is no reason why we should say this is OpenStack, this is not OpenStack. So we moved from a very product-oriented definition of what an OpenStack project is to a very community-centric definition of what an OpenStack project is.
08:00
And that has served us quite well until now. So the meat of the matter, what Python brings to OpenStack first. So I was there at the beginning when we picked languages. And I think the reason six years ago where we picked Python originally for OpenStack,
08:20
the major reason was that it was pretty important for us at the beginning that we would get packaged into distributions. I had past experience, I worked for Canonical before and we packaged Java software for our cloud offering back then. And Java is really poorly supported in distros.
08:43
While Python was extremely well supported, still extremely well supported, it was the major language for development within Canonical at the time. And so picking Python gave us a step up with distribution integration. We didn't have to fight against the distros that they would support us.
09:01
It's also extremely easy to read, which means there are not so many ways you can write stuff and so there is no endless bag shedding on exactly the format of your patches, et cetera, et cetera. It's easy to learn. My 10-year-old daughter is learning Python right now
09:21
and she's quite successful at it, so it's really easy. It's easy to patch because whenever you run Python code, you have the source code. It's not as if, when you run a Java stack, you end up with jars and you have to download or look at the source code separately, but anyone running OpenStack can actually look at the code that is running,
09:42
see potentially a mistake about somewhere and propose a patch, it's really easy. And finally, the last thing was ops familiarity. The target user, direct users for OpenStack are the operators that would deploy it. And so picking a language that was something that they were familiar with
10:02
was really helping us as well. So they are, I'm fine, thank you. Their operators are using Python internally already and so that also gave us a step up with reaching out to our users and closing the feedback loop was a lot easier, thanks to that.
10:26
So Thierry mentioned that we chose Python six years ago. We've had discussions recently about changing, but we have found a lot of reasons why Python is still successful for us. As an integration language, it's very common to,
10:42
especially in our project, to need to interface with lots of different devices or other services. Most of the OpenStack services are actually built on top of other tools that have already existed or written in other languages and Python makes it really easy to communicate with all of those things. Part of the reason for that is the large ecosystem of libraries
11:02
that's available for Python from the package index and that just continues to grow every day. So we find that we don't have to write all of the source ourselves in order to build things. It continues to be easy to read. So six years in, we have a lot of old code
11:20
and it's very important for new contributors to be able to go and look at that old code and understand what it's about and Python's readability makes that easier. Six years in, we continue to have a lot of new contributors joining our project and so having a language that is still easy to learn continues to be an important reason to use it
11:43
and we'll talk a little bit more about that in a bit. And then the community itself. The inclusiveness gives us good examples for subgroups. So PyLadies is a great example. We have Ladies of OpenStack group that's organized in a similar way. We have folks that go to conferences
12:03
and join meetups and that sort of thing which are all very welcoming to brand new members. The example that the Python community sets for welcoming new members is really important to us. So we've talked a little bit about why we think Python is great for OpenStack
12:22
and I wanted to talk also a little bit about why OpenStack I think is a good thing for the Python community. There are three main areas for that. Funding, contributors, and quality. A lot of us don't like talking about money. As we saw this morning in the keynote though, it is important.
12:40
We're all trying to make a living doing this and OpenStack gives us a good opportunity to be paid to do open source work. Often the company's funding network have goals in mind but there is still quite a bit of freedom in what you can do and the way the community itself is organized lets you take advantage of the fact
13:01
that there's a demand for that. So OpenStack companies, when they're looking for new employees, they're typically looking for one of three different key areas. Either some previous experience doing cloud or infrastructure automation of some sort with DevOps tools or existing cloud systems.
13:24
They also look for specific technologies like maybe you're a storage expert or you have expertise in networking or virtualization or they look for Python expertise because Python is a key component of OpenStack.
13:42
So this is some job information. This is trends for job listings containing keywords, Python, OpenStack, the combination of the two, storage, networking, and virtualization. And this is US data because that's what I had available but OpenStack is an international project
14:00
and I would expect the trends to work in a similar way. And this is showing growth in those job listings over time. Things like storage, networking, and virtualization and actually Python by itself are all of those lines along the bottom that are relatively flat.
14:20
Some of them even go down a little bit over time. But if you look for OpenStack jobs, that's the orange line in the middle that is trending up. And if you look for OpenStack and Python together, that's the green line at the top. So there's definitely an increased demand and that's 2100% growth over that time.
14:45
That growth in demand has the expected impact on salaries. OpenStack salaries are 17% higher than standard Python salaries. But even Python salaries, just standard Python jobs,
15:00
the salaries are trending up as well. That I believe has a lot to do with the demand from the increased use of Python within the OpenStack community. Other than a salary, there are a lot of other benefits to being an OpenStack contributor.
15:22
You are valued as a member of the community for your contributions that you give in your name, not just by the employer that's providing funding for you. Your influence is based on your reputation and so that means that you have job mobility. So we have a lot of members of the community
15:42
have moved jobs while staying within the same place in the community but they're working for a different employer. I myself am on my third OpenStack job right now. So it lets you find a company where you feel like it's a good fit, a good size and a good role for you. If you are interested in product-based companies,
16:01
there are lots of those. If you're interested in service-based companies, there are lots of those. Large enterprises, startups, you name it, there's an opportunity there. And it lets you find a fit for you both within the community and within an employer.
16:21
That influx of Python developers, folks who are joining OpenStack to work on OpenStack, means that they're also learning Python, they're also joining the Python community. So we have, Thierry mentioned Java earlier, we have a lot of folks who are joining our community coming from more enterprise situations
16:41
where they are used to programming in Java. They're learning Python which means there's a lot of opportunities to sell training to those people. They are also, many of them for the first time, learning about open source. If they're used to working on closed source applications, then they don't know about open source communities, so they're learning about that and getting excited about their jobs again.
17:03
And joining the Python community, coming to conferences like this one, going to meetups in their local or regional area, joining the Python Software Foundation, and then contributing in other ways. So outside of OpenStack, contributing to other Python-based projects.
17:23
So another way OpenStack contributes to Python is quality, and by quality I don't mean the quality of OpenStack software, I mean quality of insurance that you can get for Python libraries out of OpenStack using them.
17:41
So this is a number, and I don't expect you to guess what that means. It's actually the number of test jobs that OpenStack runs on a given typical work day. That means we run a test every four or five seconds, and those tests actually take around one hour to run,
18:00
and they deploy OpenStack software, they run functional tests, et cetera, et cetera. So what that means, we are using new Python libraries as they go out. We automatically integrate them by the time they reach our PyPy mirror, then we'll start using them. That means if a Python library new release breaks everyone,
18:23
it will break us really, really fast. And so there are lots of times where we detect problems before anyone else. The other interest in having that kind of number of tests is that we also catch corner cases and race conditions.
18:41
If something is working 99, but 99% of the time, we will hit it, like several times a day. And so it will break us too. And we, so we can contribute bug reports especially on new releases that break us really, really fast. And that's not just talking about it. I have a few examples here.
19:02
Pyman Cache 1.3.0, if you're in the room, it's okay. That was released on a Friday, July 24 last year. It moved exceptions around in a backward incompatible way, so if you were catching exceptions, then you get screwed. The bug was spotted on Sunday, and then we reported that,
19:22
and the new release was released on the same day. That's a classic release regression, and we caught it really fast. Qsu 2.1 had a deadlock when acquiring a lock that was already acquired, which is kind of a corner case, but should have been in their functional test suite.
19:41
The bug was reported the next day, the day after the release, and they fixed it just the day after. And finally, netddr0716 failed on validating a previously valid MAC address, corner case again, and we reported it the next day, and we got it fixed.
20:04
So the amount of testing we're running within OpenStack helps getting regressions detected really fast on Python libraries, because we're using a lot of them. All right, so Thierry gave a couple of different examples
20:24
of dependencies that we have. We actually have over 300 dependencies in our global requirements list for the different subprojects of OpenStack to use. And so part of what we wanna do, not just reporting bugs, but we also wanna contribute code
20:41
to those code bases as well, either directly or indirectly. We directly create libraries that we hope that other people will find reusable as well. So these are just a few examples of recent releases that we've had. Different libraries with different, from command line frameworks to testing libraries
21:02
to packaging tools. If you saw the previous talk about Haskell where he was talking a lot about making packaging more declarative, the PBR library is something you might wanna check out. That's a good wrapper on top of setup tools from new declarative packaging. So in addition to our own libraries,
21:21
we've built some tools that you don't necessarily need to be a Python programmer to find value in. So Jenkins Job Builder is probably the most popular one of these. It lets you use YAML files and templating syntax to deploy copies of jobs with slightly different parameters,
21:40
and it automatically updates Jenkins. So it's very common to see that set up with a commit hook so that when you check in the new YAML file, your Jenkins server automatically updates itself, which makes managing your test systems really easy. And there's a bunch of other testing related tools and release notes tools and things like that that are usable no matter what programming language
22:02
you're writing your code in. And then we do contribute upstream as much as possible. So I did a, just in preparation for this talk, I did a really quick survey of our community, and I had 20 or 30 different packages that were suggested as things that people contribute to.
22:23
Mostly there was evidence for each of these. It ranged from bug reports to bug fixes to new features to documentation, all sorts of things like that. And the range of all of those tools is pretty broad given the number of dependencies that we have.
22:40
On occasion, we've even taken over libraries where the original maintainer said that they were no longer interested or able to support it, but we still needed it as a dependency until we could migrate onto either a newer tool or replace it entirely in some other way. And so these are a couple of examples that I was able to find where we'd done that.
23:01
Each of these tools is a key component in one part of OpenStack somewhere. Most of them are actually used in multiple components, and so it was important to us that they continue to receive bug fixes, and in a couple of cases, security fixes were important.
23:22
All right, so we've talked about things that we're already doing to work together. We also wanted to highlight a few things that we should do to improve the symbiosis between the two projects, and Thierry's gonna start by talking a little bit about things the Python community can do. All right, so one of the things you can adopt
23:41
from OpenStack is to do actually some testing before you commit. In OpenStack, we can't really commit anything that doesn't pass the tests. That's a good pattern. A lot of people do it, but some others don't because they have commit rights, they can just push things, and yeah, you will test around release time,
24:01
but then you get around to do it. So having some kind of an automated framework where you can only commit things that pass tests, and then whenever you have a regression, you just add the missing tests so that you don't get caught by the same issue next time I think is a really great pattern. There are lots of regressions in libraries.
24:20
We produce libraries. We also regress quite often, but that really helps getting them under control. The next tip is to, if you're a library author, you should really use semantic versioning. So the idea is to signal the type of change
24:43
that's been happening since the last release. If you only fixed bugs, you just increase the patch level. If you change behavior or add a new feature but did not break backward compatibility, you just increase minor, and if you break backward compatibility and that happens,
25:02
then you increase major. It is simple, but a lot of people don't follow that, and it really helps the users of your libraries assess the risk that they are exposing themselves to by upgrading and the amount of care that they need to apply before they use your release.
25:22
So sometimes it's counterintuitive because you will increase the major version number for a security bug fix because the only way to fix the security issue is to break backward compatibility, so you end up with a big bump, but from a downstream user perspective of your libraries, it's really helpful to know that if you upgrade to that version of the library,
25:41
something might get broken versus there is no chance something gets broken because it's only a patch bump. Next one, once it's out, it's out. If you put a broken release out, the solution is not to delete it from PyPy and pretend it never existed.
26:02
It may look like the easy solution and to a very embarrassing situation. It happens to everyone, and we do quite a lot of them as well, but you really can't recall the message. Once it's out, it's out. The cat is out of the bag. Someone on the internet will have downloaded it.
26:22
Some PyPy mirror or cache will have a copy of it, and so if you remove it and pretend it never existed, then you've made an embarrassing situation embarrassing and confusing, and so that's not really, rewriting history is not giving you anything. You should just roll forward,
26:40
release a new version that fix it, and in your project documentation history, just tag that version as unusable, and that's fine. It's the only way to really make progress in a constructive way. Last piece of advice, don't release on a Friday, and by Friday, I don't really mean Friday.
27:01
Just don't release before you go into an extended vacation or last hour of the day and go to sleep because for the previous, due to the previous points, you will release regressions. That happens to everyone, and so you kind of need to be around to fix it and get early feedback from people
27:20
that can report that it's broken, so wait until Monday. Just do it in the middle of the week when you still have a few days ahead of you rather than do it as the last thing before you disappear, but there's also plenty of things OpenStack can do to make pattern better.
27:43
Right, so the first of those is contribute even more upstream than we do now, both in terms of creating new things that are more usable than what we have, but also contributing to all of those 300 plus dependencies that we have, and not just fixing bugs, but also writing new features,
28:03
so we frequently write little wrappers around libraries to do extra things that we feel like we need to do. A lot of this could really be contributed to the library itself, to make either wrapper APIs that are part of the native library, or just extend the existing API in some way to make the library more featureful
28:21
and give those things to everyone. All right, we've become a little bit better about creating reusable components from OpenStack. A lot of them are really only reusable within other OpenStack projects because of the nature of some of their dependencies. They sort of expect to be running inside of an OpenStack framework in some way,
28:42
but I had a list of six or so libraries that we've created recently, and there are actually a few others that we have that aren't well documented yet that I could have mentioned on that slide, so we need to do a better job of finding places where our code is not so special and snowflakey
29:00
that we can't give it to other people. And then highlight the work that we've already done, so I mentioned that we have some things that are not well documented. We should fix that, work on the documentation. We do have a bunch of documentation for our libraries up on docs.openstack.org slash developer, so if you are interested in taking a look at what we have,
29:22
all of the documentation is hosted there with our other documentation. And then we should also be doing some work to share the tools and experience that we have by coming to conferences and speaking more often, so frequently it's pretty common to have folks come
29:41
and talk about OpenStack itself, but there are a lot of aspects of the work that we are doing that is not specific to OpenStack and experiences working on large open source projects at scale, deploying things at scale, automation, things like that, that we could talk about using OpenStack as an example, but not requiring that you care about OpenStack in particular
30:01
to make use of that stuff. So if you are interested in learning a little bit more about OpenStack, there are a couple of other members of the community who are presenting this week. And Terry, you had something else you wanted to add? Yeah, we actually have the OpenStack Summit every six months and it's going global everywhere.
30:23
It comes to Europe every two years, and this year in October, it's in Barcelona in Spain, so if you're local, wanna join, it's a great experience to be around and it's actually free to attend if you contribute one patch to OpenStack, so you might be interested in doing that.
30:41
And if you have any travel assistance or anything, we also have a travel support program to send people that don't necessarily are paid to walk on OpenStack and are interested in learning more so that they could attend the event. So if you're interested, you can also join us in Barcelona because it won't get in Spain until at least two or four years from now,
31:02
so take the opportunity if you can. And that concludes the talk, I guess. Thank you for joining us. Plenty of time for questions.
31:21
Thank you to the organizers for providing those awesome orange towels that match our slide theme. That's what we need. Question? Okay.
31:41
Hi. When do you think OpenStack's gonna move over to Python 3? Great question. We have been working on that for a couple of years. Actually, I've been involved in OpenStack about four years now and we've been working on it
32:00
at least that long. The process has been a little slow. So the first phase, well, first of all, I should explain that it is, given the size of the contributor base and the amount of code, we try not to do things big bang. So we try not to say, we're gonna move everything to Python 3 right now.
32:21
So we phased it into a couple of different steps. So the first phase was to make sure that we were only using dependencies upstream that supported Python 3. I think eventlet was one that it took a little while to get a Python 3 version that was stable for us. There were a couple of others that that was sort of the big one where we knew we really didn't even need to bother with anything else until that was ready.
32:43
And then the next phase was to make sure all of our internal code, our internal libraries that we manage were ported and that step is done. We have most of the projects running unit tests under Python 3 now. So they've run under Python 2 and Python 3
33:00
and we're at the step where we can start running our functional tests and so projects are starting to set up their deployments for the continuous integration tests to run under Python 3. So one thing we do is that once you get the unit test to pass with Python 3, we run tests for Python 2 and Python 3 so that you can't really reintroduce a regression on that
33:22
and so we support both. At some point, we'll probably only support Python 3 and go forward but the idea is to support both first and like Doug said, we already have libraries and I think like 80% of the, is Victor here. I think we're down to Nova and Swift.
33:43
And that's the work is coordinated by one person, Victor. Okay, another question for you. Hi, Thierry, hi, Doug. So there's been a lot of discussion lately in the community about adopting Go because it's faster for some stuff.
34:01
How is the community facing this problem in terms of feeding back the limitations of performance kind of code to the community in particular with the async kind of code? So it's a bit of a complex discussion to have because on one hand, you wanna give developers
34:21
the tool that they need to do their job. On the others, like I said on the slide about the big tent, we basically define OpenStack as being a single community and so it's really difficult to fragment it around language lines. Even if Go is not the worst language we could work from, work with coming from Python,
34:41
it still will create fragmentation between the part of the community that only does Python and the part of the community that only does Go and people that do both and so it's a bit of a struggle to try to like enable developers to do it without ruining our community.
35:00
The way we've been looking into that is every time someone has a need for Go in OpenStack, it's usually that they have some kind of a specific problem that we kind of need to look into and for example on a recent one, people assume that Go will be faster for async programming and very fast network outputs
35:21
and the recent progress on AI, or HTTP, and other libraries in Python shows that it's not necessarily true and sometimes it's education, sometimes people just want to jump to the new shiny thing that everyone else is using so we need to kind of debunk that and if people still have a reasonable use case for Go,
35:43
we need to find a way for them to use Go. They could also develop a Go library in a corner and call it from Python code. They could, I mean there are other ways to allow developers to use Go without fragmenting the community and currently we're basically looking into
36:00
how we can unblock developers without ruining the community. And as far as upstream work, Larry Hastings is giving a talk about the Gilectomy that he's working on to remove the Gil and I think that was one of the big performance issues that one particular project was seeing. He's giving a talk on that later this week
36:21
so I'm looking forward to that. And we do have a couple of members of our community who are Python core developers and they're well aware, Victor Stinner especially is well aware of the async issues and expect all of that stuff as being fed back upstream that way. But we're sort of at the point
36:43
where we have a particular project where it is very clear that there are low level issues dealing with hardware devices for their performance. And as Thierry said, there may be architectural changes to that particular application that are needed in order to split it up.
37:00
But everything that we do, we've been trying to push back upstream as far as issues and performance. I've filed a couple of bugs myself. I know some other members of the community have too. Okay, thank you. Last question for you. Along the same line of the last question, you briefly mentioned that you were thinking
37:21
about replacing Python somewhere in the past. What were the languages you were thinking about as contestants to Python? Yeah, so we're not thinking about replacing it wholesale. There are a couple of projects that have run into particular performance issues at very, very low level dealing with storage devices.
37:44
And those projects are experimenting with writing things in Go. I'm not sure whether they evaluated any other languages. No, and we also had a number of projects that wanted to have a project written in Java,
38:01
but it was more because they had loads of corporate developers that were Java developers somewhere, and they wanted to reuse them for OpenStack without training them to a new language. So there was no technical reason. It was more like a business reason. And so that's the two things. At one point, there was Erlang at the very beginning.
38:21
People wanted to write a message queue, and they said that Erlang would probably be the best thing for it. So every now and then, someone has a very specific use case that they want to use a very specific language for. And our role as the technical committee for OpenStack is not necessarily to give direction for every project to follow. It's more to make sure that we still have
38:41
this coherent community and follow those principles that we've set. And so the language is a thorny issue because of this potential for fragmentation versus the need for technical abilities. Like Doug said, Go is the big thing right now, and people want to rewrite things in Go.
39:00
So we need to have them pose and look at the problem really deeply before we can actually say, well, Go is the only solution for you. Python can actually be made to work for you probably, and you should look into every specific issue in detail before you switch. And we do use plenty of JavaScript for front-end work
39:22
and bash scripting and things like that. So it's not that everything in OpenStack is written in Python but most of the application stack is written in Python. Okay, thank you for this interesting conference. And thank you very much to you for coming. Thank you. Have a good night.