So thank you very much to the organizers for having me and for sorting out all of our technical problems. So as was just said, my name is Nicole. I'm a designer and a web developer. I am Australian, as mentioned, but I do live here in Scotland. I live in Perth, which is a beautiful part of the country. It's about an hour north of us here in Edinburgh.

And just to make things slightly more complicated, I actually work for a French company. So I lead the UX and UI team at PeopleDoc. We're actually a sponsor of EuroPython this year, and we're hiring. So if you're looking for a position, come and talk to me. But that's not why I'm here today to speak to you.

I'm here instead to talk about my experience working on Warehouse, which is the project that currently powers PyPI, not PyPy, PyPI. We have to make that distinction for obvious reasons. So I've been working on the user experience, user interface, HTML and CSS code base on Warehouse for about three years

now. And via Warehouse, I am a member of the Python Packaging Authority. So that's a group of developers who are generally focused on improving the state of the Python packaging world. Now, a fun fact that I found out whilst doing some research for this presentation,

the PyPA, one of the original proposed names was the Ministry of Installation, which I really love that. I'm quite disappointed they didn't choose that in the end. But it so aptly describes really what the PyPA is about. It's about installing stuff, basically, on your computer. And also via Warehouse, I'm a member of the Python Packaging

Working Group. So the Working Group is an organization that's a sub-body of the Python Software Foundation. And our goal is to raise money to try and improve the state of Python packaging. So the long-term vision is to be able to fund both PyPA projects,

so the official projects, things like PyPI and PIP and Virtualenv, but also to be able to have funds that are available to the community to be able to fund different projects that are emerging in the packaging space. So as described, I'm here today to tell you a little bit about the Python Package Index, its story,

look at its history, and also ask some questions about where it will go in the future. So basics first. A quick introduction for newcomers. What is the Python Package Index? So this is my definition. I didn't get it off Wikipedia or anything like that. So it's the place for Python programmers

to publicly share their code so that other people can use their code. So when I say, I've deliberately highlighted the, because it's the place that's supported by the PSF and recommended by PyPA tools. And publicly, because there's obviously lots of other ways that you can share your code, but this is the place

where the community has chosen to invest. So it's built for the community, by the community. And you've probably, even if you don't really know what it is, you've probably already used it, because when you type pip install my favorite project, what's actually happening is you're getting the file off the PyPI servers.

And it's mostly via pip that the index last month served 11.2 billion HTTP requests. And if we extrapolate that out over a 12-month period, it means that we're serving about 134 billion, 400 million HTTP requests a year.

That's probably a conservative estimate because that's going up every month. So this is basically what we're handling. And the other side, other than obviously being able to access it by pip install, is we have a web interface at pipi.org. And that's the place where you can go and you can search for different packages to install

and you can find out the information about packages. And on pipi.org last month, we had 1.3 million unique visitors from 228 different countries. And if you start to break that down by region, we can see that the largest group of users is actually located in Asia, followed by the Americas,

so that's North and South America bundled together, followed by Europe, then Africa, which has got a growing Python community, and Oceania. If we break that down by country, you can see that the US is just ahead of China. China's rapidly catching up with the US, followed by India. And then in terms of the European community,

in the top 10 we've got Germany, the UK, Japan, and Russia. Sorry, not Japan, France and Russia. I just relocated Japan. So all of this costs a lot of money. So it costs, this is a back of the envelope calculation, costs about $118,000 US a month to run PyPI

in terms of servers, CDN, monitoring, paying for search, that kind of thing. And all of those services are currently donated by sponsors. So again, if we extrapolate that out to over a 12-month period, it costs about $1.4 million a year to run the index.

So all of that to say that the index is big and it's important. So I joined Python Packaging about three years ago. I'm quite a new face on the scene. I found it really interesting to take a look into the history of PyPI to see how we actually got to this kind of mammoth project.

So this is what I found. So I want you to cast your mind back to the early 2000s, if you can. I realise there will be some people in this room who were very young infants at the time, perhaps not even born. It was a time of Tamagotchi. I remember these at my school, amazing.

Double denim. Hotmail was the most popular email service because Gmail didn't actually exist at that time. Google did exist, but it looked like this. But you might have been using Ask Jeeves anyway, if anyone remembers Ask Jeeves. And despite Python being released for 10 years, the Python Software Foundation wasn't actually founded until March 6, 2001.

So at that time, several developers had independently released and they recognised the issue of distributing packages. And several independent indexes had popped up. This is the vaults of Parnassus, which I think is the most amazing example of early web design.

And this was an absolutely, it was probably the most popular independent index at the time. In 2002, so about a year after the establishment of the PSF, Richard Jones proposed PEP, Python Enhancement Proposal,

for those of you that don't know, PEP 301, which basically proposed a central index server. So something that was going to be hosted on the python.org domain, the PSF official property, that had an air of legitimacy. So the idea to be able to combine all of those are previously sort of scattered approaches to establishing an index.

And that's basically what happened. So we know that PyPI was launched in very late 2002 because we have record of four projects registered. So my bet is like late December, someone hacking after Christmas or something. But in 2003, that's when the project really took off.

So we had 273 projects uploaded in 2003. At this stage, there was no data actually hosted on the index. So basically, it was just a list of projects. You couldn't really even search it. You could find things by classifier, so kind of like tags.

And the workflow for people was to go to the index, find project that they're looking for, click through to the readme, and be able to download files directly off the readme, which were hosted on somebody's server somewhere on the internet. Or you could also go to the homepage of the project and then find the files that way.

So a very manual process. To give you some context around what was happening in the Python community at that time, so in 2002, there was the first EuroPython with 240 attendees. And in 2003, the first PyCon in the US was hosted with 200 attendees. So really, the Python community in its infancy is starting to grow

and starting to contribute to the index. So 2004 comes along, and we have EasyInstall. So that project was kicked off in March 2004, and basically, it tried to automate the process that people were doing manually. So it would go to PyPI, basically crawl around

and try and find links of things to download onto your computer. Or it would follow links to try and find things to download on somebody's website somewhere on the internet. In 2005, file uploads were added to PyPI, so people didn't need to host their own files. It could actually be hosted on the index.

So this is what the index was by 2007. In 2007, we had 1,249 packages uploaded, so quite a substantial growth compared to a few years before. And more and more of those projects were actually hosted on the servers rather than being hosted in somebody's server in their cupboard.

And this is what the index looked like, and I think many of you will recognize this because from a web UI perspective, nothing really changed very significantly in the next 10 years. It still looked pretty much like this by 2017.

But meanwhile, in the background, whilst nothing was happening at the front, the popularity of Python as a language really skyrocketed. And as a result, the popularity of PyPI also grew. So this graph, that graph, shows the growth in the number of packages

added to the index year on year. So that's not the cumulative number of packages on the index, simply showing the growth. Obviously, cumulative would be a much steeper curve. And this obviously put a lot of pressure on the index in terms of its infrastructure. So behind the scenes, developers were working really hard

to put out fires and scale the system for the legitimate use of the index from the community. But also, as the index became more popular, there was more malicious attacks, malicious packages, and spam that needed to be dealt with as well. So I have a picture or a rare footage of a core dev at that time

trying to work on the code base. Basically, there was a lot of firefighting, because by this stage, they were working on a code base that was, if you look at 20, well, 10, growing in age. Let's say it's growing in age,

and sort of not designed for the kind of scale that it was handling. So in terms of scaling, this is a very brief overview of what happened. So the original code base assumed that PyPI was hosted on a single server, and that's exactly what happened.

So it was hosted on a server called Dinsdale, which was located in the Netherlands, with a company called XS4ALL. So just sort of standard hosting. In late 2012 or 2013, I couldn't quite get the date for this, the infrastructure was moved to the Oregon State University Open Source Lab,

and DRDB was put in place. So that basically created mirrors for disaster recovery. In 2013, Fastly, which is a CDN, was added, which is basically caching on steroids. They can contact me for that catchphrase later. And in January 2014, probably the most significant change

in terms of the architecture was that the code base was moved across to Rackspace using GlusterFS. So if you are like me and you're not an infrastructure person and you don't know what GlusterFS is, basically the very, very high level summary is that it's a clustered file system,

and it means that PyPI was running on many servers that our team could access as though it was a single unit. So it gave the team the opportunity to be able to spin up and retire, shut down more resources as and when the project needed it.

At the same time, I should say, there were several PEPs also proposed to help try and improve the consistency of the Python package index. So even though we'd sort of scaled the index, there were still problems that remained. First of all, PyPI predates almost everything on PyPI. So I think I said 243 packages in the first year.

It was built really before we knew how to build great web applications with Python. So we didn't have any modern web frameworks to work off. So the code base is using custom code, which means that it's really difficult to maintain.

And I think I never personally maintained it, but the sort of horror stories that I've heard certainly indicate this. And because it was built in a technology that's not that well known or not a popular sort of web framework, it was really difficult to attract new contributors to the project.

Also, it was really difficult to set up. So Donald, who I work with on PyPI, told me that he used to have to comment out certain parts of the code base just to get it to run locally on his computer. So kind of a real nightmare for attracting new contributors. And because of that, I think no is a bit harsh.

So no significant new feature development, which means that in terms of features, it kind of stagnated. Also, because it was difficult to attract new contributors, we had poor bus factor, which is a rather brutal term for saying what would happen to your project if one of your core team members were to be hit by a bus.

My preferred version of that is that they leave on a bus. But anyway, very poor bus factor. So you had really a handful of people who knew how to fix problems on PyPI when problems inevitably occurred. So here enters our hero,

at least one of our heroes in the story of PyPI, and that's Donald's staff, who I have a great deal of respect for. So Donald saw PyPI and saw the problems with PyPI and decided to have a go at doing something different. So in 2011, he created Crate, which was an alternative service to PyPI

that was using the data on the PyPI servers. And I'm actually going to directly quote him here. He said it was a bit hacky, but it was popular. Shortly after, he started making commits directly to the real PyPI code base. And in 2014, he decided to shut down Crate

because he was sort of duplicating his effort across two projects. At the same time, he was constantly thinking about how could he rewrite the PyPI code base to make it easier to maintain and to contribute to. So he did a number of proof of concepts during this time. And in 2015, something stuck.

So the version of Warehouse, which is the project that I work on, was established in early 2015 using the Pyramid web framework. And that's where I come in. So in June 2015, Donald posts an issue basically saying, I'm really bad at design.

I need some help with this. And through a good friend of mine, I find out about this. And that's when I start to become involved in Python packaging. So, Warehouse, what is it? So as I said, it's using Pyramid. So it's using modern tech stack. So we're using Pyramid. We've got Elastic for search and SQLAlchemy as our ORM.

We've got really modern tooling as well. So it runs on Docker. It's really easy to set up with Docker Compose. We've also got continuous deployment, which is really fun now that it's being launched because if you make a change on the code base, you can see it live on this massive website within about 15 minutes.

It's more stable and more secure. I hope it's got an improved user experience. Otherwise, I haven't done my job very well. And it's easier to contribute to. So as a project, we work really hard to support new contributors to Warehouse. We've done a lot of work on our documentation. We try to support people through the pull request process.

And one of the things that I'm really proud about on the index is that we've had a number of people who've made their first open source pull request on the Warehouse project. So many people might think, oh, it's a big project. I couldn't possibly contribute to something like that. But actually, you can, and we really want you to.

So throughout the development of Warehouse, I've always been, well, I think everyone's always been, really optimistic about when it would go live. This is a story, rather crushed because of the VGA port. This is a picture of me presenting at PyCon France in October 2016. And you can see in the corner there, it says,

what it actually says is good news. We're almost ready to release. So that was optimistic. And I think that kind of optimism went through the project for a long time. But in reality, we had some problems with trying to get the Warehouse code base live. So first of all, the speed of development was slow because we were relying on community contributions

and everyone was kind of stretched working on other things. There were many major features that had not yet been started. So for example, the area you log into to administrate a package hadn't even begun. And I think I completely underestimated how much work that was going to be. We didn't have a release date in sight.

We had no real project management around the project. And to top it all off, the old code base, people were still firefighting on that, which was taking away resources from being able to get Warehouse live. So in 2017, the Python Packaging Working Group, another hero of our story,

applies for and receives a 170,000 US dollar grant from Mozilla. So that's the Mozilla Open Source Support Award, and that was awarded under the Foundational Technology track. So actually, I'd like to give Mozilla a round of applause for that if we can.

I really, I think the whole team really appreciates the support that we've got from Mozilla to be able to work on the project. So this award was specifically to fund the development team to bring Warehouse up to feature parity with the old code base.

And to release Warehouse and to shut down Legacy Pi PI. Through the grant, we funded myself, two developers, project manager and her assistant, as well as our PSF liaison. So we had basically a team of, what's that, six, six and a half people working on the project. And we achieved a lot with that money,

and within the five months that we spent that money. So we worked on authentication workflows, account administration, you know, you can reset your password, which is a good thing. The management of projects, releases and files. We've solved a lot of UI problems, lots of bug fixes.

We added a lot to the documentation. So one of the things that I'm really, really proud of on the new Warehouse is that we have a great help section. We linked through to the help section. We linked through to a lot of documentation. We found that really important because we want people who are new to the Python community. We know Python's being used in teaching a lot.

To be able to understand what the Python package index is, because it might be the first time they've actually used any kind of index. And we did an infrastructure overhaul. So that was to support Kubernetes-based continuous deployment, end-to-end encryption and secrets management.

During this time, we worked really hard to build the community. So we merged 425 pull requests. We closed 302 issues. We supported 26 new contributors within that five-month period. And of the 425 pull requests, 149 came from the community, which is about 35%, obviously excluding bots.

Unusually for a software project, we came in both on time and within budget, which I think is really a credit. Thank you. I really have to credit Sumana from Change Set Consulting,

who is our project manager, who did an amazing job of hurting everyone towards that outcome. So in March 26, we released our beta. We didn't have too many problems. So by April 16, we went live with the new code base. And by April 30, we turned off the old code base. And there was much celebration.

Donald had actually stockpiled a whole number of animated GIFs to post on Slack for the very occasion. So it was just celebration everywhere. And I think the new code base and the new website is awesome. That's my humble opinion. I might be biased.

So this is what it looks like. So that's my work. And for our users, it's got a lot of really great new features. So we've got markdown support. So what you can do is, if you're wanting to enable markdown support on your project, Read Me, you can search markdown description on PyPI, and it will come up with an example project. And that has all of the instructions

on how you can get markdown support, which has been a very heavily requested feature for a long time. You can get that now on your Read Me's. It's got a vastly improved search from Elastic. At the moment, we're not doing anything too fancy, but because we're using Elastic search, it means that we can really extend

the search capabilities of the index, which is really important. It's fully responsive. So if you wanna check out packages whilst you're using your mobile phone, you can. And you can also do all the administration tasks on your mobile as well. Lots of help resources. As I said, it was a really important part of the design for us.

And we've also got a chronological release history, really easy to see in terms of the history of an individual package. So that's what we've got for you. But for us, we've got some pretty important things too for our own sanity moving forward. It is scalable. It is extendable, learnable, and maintainable. So it's got those four things that are really, really essential

towards the health of the project ongoing. And we've got great ideas about how we could extend the index. So one thing that I would really like to do is to make the user interface more accessible. The Python community, as we all know, is really welcoming to different people. And my opinion is that we should not restrict people

who have disabilities from being able to fully experience the Python package index. So we really want to make some accessibility improvements on the front end code. We've had an audit on that, and there's a few points. In fact, we can talk about those at the sprints. As you saw earlier, most, well,

the largest group of our users are actually based in Asia. So it would kind of make sense to be able to localize and internationalize the index. I personally would like to do a lot more design research and UX improvements on the index. So right now we know it's not perfect, but it's hard to make decisions on how to improve it because it's difficult to know what people want.

So recently I did some design research where I asked people to rate what's most important to them on the project detail page. That data was really, really interesting, and it's linked to in my slides. And I'd like to continue to do that kind of work to really understand what people want and what people need from the index

so that we can improve the design in line with those needs. We'd like to add two-factor authentication. There's currently a specification in progress for this. And we'd also like to improve our audit trails. So right now we know about, so we have a project journal basically, so we can say what happened on an individual project.

So someone was added as a collaborator or there was a new release made or a release was deleted, for example. And we were collecting things like IP address and date, and that's about it. And what we'd really like to do is to collect more information on sessions, et cetera,

third parties that maybe are interacting on PyPI, and to be able to, if something goes wrong on the index for our unsafe, to be able to see what happened, but also to expose that information to the users so you can keep an eye on your PyPI account to see if anything fishy is going on. But my question is, how are we gonna get there?

And I've got a few questions to propose to the community about that. So the first is, do we as a community value the service provided by PyPI, and how much? So would we be able to fill the $1.2 million gap if our sponsors were no longer able

to provide infrastructure donations? Do we care enough about the project to actually pay people to contribute to it? As Python continues to grow, is it sustainable to rely on a handful of people to maintain and improve PyPI? Now, I'm not underestimating the number of people who contribute to the project,

but at its core, there's a handful of people who are steering PyPI in its future direction. Now that the Mozilla Open Source Support Grant is over, we sorely miss the time allocated to project management, community engagement, and feature development. And we have had actually on the project

123 amazing contributors, but it takes work to support all of those people. How should PyPI evolve to meet the changing needs of our community? So obviously the Python community is growing, growing in its size, in its complexity, the different ways that people are using Python.

Also computer science is evolving, as is web development. People expect different things from their services. So no doubt, people are going to start to expect different things from the Python Package Index. Are we actually equipped to deal with this? And if not, are we prepared to allow commercial interests to fill that gap?

So, how can you help? This is my call to action. First off, can you please verify your email address? I don't know if you know about this, but basically at the moment we're sending a lot of email to unverified email addresses, which means that our email address can be classified as spam.

So that's not a great situation to be in. So if you've got an account on PyPI, please go and verify it. It takes two minutes. You can engage with us. So dist-util-sig, that's the mailing list. We've got an IRC channel, PyPI, that's on Freenode. And you can engage with us on our issue trackers. So it's not a really scary world.

We really wanna hear from you. We really want you to engage with what we're doing. You can contribute to our house on GitHub. That's the address there. And you can sprint. This weekend I'm running a PyPI sprint at EuroPython Sprints. So I'd really, really love for you to join me there. We've got a whole number of issues that have been specifically tagged with

ready for the sprint. So I'd love to have your contribution. You can donate or you can ask your company to donate at donate.pypi.org. So we've recently added support for recurring donations and any recurring donation would be most welcome because that would allow us to actually make plans

in the long term about how we can allocate resources to not only PyPI but other packaging projects. And you can thank our sponsors. So obviously we mentioned Mozilla earlier that paid us 170,000 US dollars, which is great. But we've also got all these infrastructure sponsors. So maybe if you're in charge of choosing

who you're going to use in your organization, have a look on our sponsors page. It's linked to in the footer and see who is actually supporting the Python community. So that's it for me. Thank you.