Okay, everybody's here. We have the final speaker for the day. It's Ricardo Banci, who's going to talk about lessons learned after 190 million lessons served. Good afternoon, thank you for coming.

I'm here to talk about some lessons. Before anything else, I have to say that I'm new to Udemy, and most of what I'll describe happened before I arrived. What I'll talk about is therefore the work

of other people, and the lessons, however, I think they stand on their own. Well, this is where we are now. Udemy was founded in 2010, and in six years it grew to 11 million students, 40,000 courses

given by 20,000 instructors in 80 languages. The 190 million lessons served number I used for the title was gathered from a table that tracks students watching our videos. Lesson one is about preserving our culture as we grow.

We're a learning company, and learning is in our DNA. For any healthy organization, preserving its culture is essential to its well-being.

What I'll talk about is about engineering team. Other teams have different things they do. Well, we are very serious about onboarding. Onboarding new employees.

The idea is to get a new hire from a new hire to a productive engineer in the shortest possible time. We use extensive automation. We have standardized development environments, development environment that's reasonably flexible.

And if you really want to, you're able to deploy our website on your first day. I did that in the first week. Even I can do it. We learn and teach ourselves using our own tools.

There are Udemy courses about our applications from the user perspective, from the backend perspective, that are given by our own team. So you can hear about the application as it's described by the people who created it.

We also communicate a lot. We have hundreds of people in three continents. And we spend 11 time zones. We all hang out on Slack.

We use HipChat until I like yesterday, literally. We just moved. We also use Scrum. We have the usual stand-ups, grooming sessions. We also have hangouts, we conduct all-hands meetings

on hangouts or other solutions if we have too many people. Every meeting room is capable of joining a hangout. We have big screen, a camera, a good microphone, so we can have easily meetings

that span more than one team in one place. Well, like I said before, we use automation a lot. We automate the quality checks. We can't commit ugly code, seriously.

We can't commit code with one extra line. We can't commit code with missing spaces. We can't push broken code. That means Git runs a basic set of tests before we are allowed to commit,

before we are allowed to push. To commit, it's flake8. So still, anyway. And you can't merge if your code doesn't pass a very extensive set of tests. So you're absolutely sure that when you merge your code, you will not break anything.

Also, we also use custom tools, custom Django admin scripts, custom Django admin commands to help prevent accidents.

So if you want to update a series of records, a series of objects in the data store, you will not be handed a MySQL prompt. In fact, it's very few people who have that kind of access. Instead, you'll do that through code that's reviewed by someone else

that's known to be, I won't say bug-free, but it probably is less bug-free than you typing SQL commands on the MySQL prompt. Even though we go to such great lengths

to prevent that, some accidents happen. I myself deleted a shared database on my fourth week at the company. I was out of the Dublin office where I'm based, I was in San Francisco. And I deleted one of our shared databases. Not production environment, development environment.

But thanks to me, the whole development team with the San Francisco office, which is very large actually, had a mandatory coffee break. Because no one could do anything. Yeah, there's some room for improvement.

What we can do to that? Well, we have the shared database. We're working now to have an individual database for everyone with just enough data so that you can run your development with meaningful data.

We also constantly make improvements to our development environment. We add, all the time we add code that makes it easier to develop. We add extra checks, we add some extra functionalities that were not there before.

Personally, last week I think, no, last month, I added a bit of code that made some tree objects represent themselves in a nicer way on the terminal on the IPDB prompt. It was a horrendously complicated logic. And by that, I had someone lessened my cognitive workload

so that with my somewhat limited brain, I could wrap it around a very complicated problem without having to worry about imagining what the objects looked like. I could just bring them on the screen and have a look into them.

The second thing, the second lesson is that as you grow, you start to, you have to, well, all the time you should be measuring everything anyway. When you grow, you will start measuring it in bulk.

So every application we have generates, collects metrics about themselves, about itself. We collect those metrics in a huge, not Facebook huge, but still pretty impressive scale.

For that, we use stuff like Sentry, Datadog, and ChartIO. This is not, although this is considered something important to us, these applications are good enough so we don't have to develop them again. These guys are good.

So, Datadog, Sentry for alerts, Google Analytics, and most of the front end is constantly monitored with that. Also, we have a huge user base. With that comes the possibility of doing extensive tests and experiments

where you enable some feature and you can track on A-B tests how the user base behaves. Since you have a lot of people coming, you'll have good quality data on that. And we can actually know how something will impact us well before we enable it to the whole user base.

This way, surprise are minimized. Which is good. I don't like surprises. That also allows us to have early warnings. If something starts to,

if you see the distribution, say, of execution time of a given function and you have a peak at like 100 milliseconds and you start developing a peak at zero milliseconds, you know something is wrong. So you can watch those distributions on a tool like Datadog.

And this data must be timely. We must receive it soon after it's collected. We must be able to see it immediately. This large user base, it's convenient because it generates a lot of data we can analyze.

But it also amplifies any trend very quickly. And it's really not that good for you if you only know what caused an outage, a week-long outage of a service.

You don't want that. You want to know if possible before. Am I going too fast? No? Yeah, I suspect you'll like this one. It's about questioning and being a bit

willing to abandon an approach. Let me say that. They, Netscape in this case, delayed a new version by three years. They did it by making the single worst strategic mistake

that any software company can make. They decided to rewrite the code from scratch. Okay, this is Joe Spolsky. Who knows Joe Spolsky? Raise your hand. Well, for those who don't know him, he's a pretty clever guy.

He created a lot of stuff we use daily, like Trello or Stack Overflow. Who is a Stack Overflow here? Come on. No, I don't believe you. No, no. Everyone should raise hands that. Okay.

He said that in an article, this things you should never do, that everyone should read, even if you want to disagree with him later. Told you you'd like it. Yes, we moved from PHP. We had a large PHP application built on a custom-made framework

created by, designed internally by our founders. Who here has built their own framework? Who thought at that time it was a good idea?

Who regrets that now? Okay. We're good. This is a problem, because new hires have to be trained on a framework they have never seen, and it takes a long time.

It ties an expensive resource. It ties an engineer that's already trained on that. And the engineer will be tied and not building useful things, useful things for the business, useful features, things users are demanding. That specific code base had very little tests.

With the lack of tests, maintenance becomes walking on the minefield. The fear of breaking stuff and not seeing it until it goes into production or worse yet, until it corrupts user data

will drive you to adopt a very careful approach. You will always, you end up opting for safe changes instead of writing some more maintainable, more elegant, more performant code.

You may even be tempted to replicate side effects of a previous implementation out of fear you'll break something you are not seeing. And obviously, this makes development slower and more expensive.

So what did they do? And I say they because I arrived right after the migration was finished. So I didn't even see that happening. I'm just such a shame, I'd love to see that. I'd love to be able to delete some PHP code.

So what did they do? No, they didn't jump for the first technology or the technology they loved the most. They actually created a project and evaluated. They evaluated the various options. They did a small proof of concept project

with each of those and with what they learned, they guided the decision. They looked into how easy it was to develop for that platform and what they knew, how well the platform encourages good practices,

and how easy it is, it would be, to onboard new engineers on them. Yes, this is a serious project. It's not, a lot of resources was put behind it.

They set deadlines, they agreed on goals. They did that as if the company depended on it, because actually it did. Well, I know you probably have guessed which technology they chose, because I wouldn't be here talking about it.

I would be on NudeConf or something, some other conference selling stuff instead of here. Well, what Udemy got? And what beautiful environment I have the luxury of working with. Well, we got a modern Django-based application.

And modern, it runs in Python 3.5 and Django 1.8. We are considering the move to Django 1.9. It also allows us to onboard engineers much quicker, because Django is an opinionated framework

and you don't have to teach every decision we made. Every decision Udemy made, a lot of decisions are already baked in the platform. We have 88% of test coverage. That's not perfect. We can do, we probably will improve that.

One of the things is that we can't commit code, we can't merge code. If we've demerged, the code coverage goes below a certain threshold. I have coerced this decision a couple of times myself.

In the end, it's for good. With all that, we had also a lot of new features. We had a more consistent code base, which is what opinionated frameworks are for. And well, there's some, of course, as anything that was migrated,

there is some scar tissue where PHP and Django co-existed. Some table names are a bit odd, some field names are a bit interesting. We have some special field types and some parts where we have to fight Django's opinions.

And let me tell you, Django is very persistent in fighting back. Who had here to fight back? Oh, you laughed, come on. Someone raise your hand. Okay. And what did we learn from that?

Well, the migration took a while. It was two years. Two years with 30% of the engineering team dedicated to it. It was done incrementally, feature by feature. And there's a motivation thing on that.

The team involved, the team responsible for the migration tracked the progress by counting the lines they actually deleted. This is why I'm wearing this t-shirt. They call themselves the Delitinators.

We've cried. Deserved one. I borrowed the t-shirt. I didn't delete any PHP line, so. All the time, the website was up and running. And it was, as you would browse the pages,

you would see a hybrid PHP Django application showing your page, showing the data you were used to. Also, since everything was changing, mindsets would have to change. So that's where the automated checks come from.

So instead of ending up with an untested code base, they enforced some things that they considered mistakes. They learned from the PHP code base and they worked against the impulse to allow the Django code base to be like that.

There's also a final lesson here. This is about the importance of the mission. As I mentioned in the start, we are a learning company.

We exist to help anyone learn anything. That's quite a goal. This also means that anyone can teach something to anyone else. We all have things we like to talk about. We all have passions. We all have stuff that we would like

to teach other people if we could. Or there are stuff we'd like to know more. If only we could find a person who could teach us. What is the impact? About one third of Udemy students are starting or growing their own business

using the knowledge they acquired through the platform for that. We also have lots of people who became full-time instructors who basically changed careers

from being something to teaching. That's something they love to do. If you visit our stand, you'll see a couple sheets of stories of how Udemy changed the lives of those people. I'll tell one of those stories,

not one that's on that sheet, those paper sheets. I'll tell the story as it was told to me by my colleague, Aaron, who actually knows this smiling, nice guy.

He's Lynn Smith. He had a lifetime career as a copywriter. He wrote ads for companies like Vodafone, Lloyd's, BBC, IBM. After this lifetime career,

he decided it was time to retire and to keep himself busy as he winded down his company, his career, he got involved with Udemy and he decided to start, no, I can start teaching people to write proper copy.

Well, it turns out he probably didn't retire because as of now, he has taught more than 56,000 students.

I myself was very impressed with his number because I was once a teacher and my largest class was like 50, 60 students for a course that took 20 classes, 20 weekly classes. If I were to get close to what this guy did

after he retired, it would take me 500 years to do that. This is a 10x guy, a 500x guy probably. When I read the stories, when I read the messages or internal emails

sent forward that our students sent, I see our mission is very, very important. In the end, everything we do, everything we do at Udemy everything we all do here at EuroPython

is about the people, is about the community. It's about people, it's about teaching, it's about learning and sharing. This is what we do. Thank you.

Oh, one more thing. In case you buy a course, if you use the import this for zero code, you'll get a 40% discount.

Any questions? Okay, give a big, okay. Thank you.

I know nothing about Django, but you said that you have to fight the framework back. Could you, an example, like I know nothing about Django. Django's an opinionated framework, so it assumes you want to do stuff in a certain specific way. I know it's no longer the case,

but a good example I can bring up is from I think a long time ago, and I met that situation. I wanted to, I was doing an image bank for a portal, and we wanted to save the images,

the originals in a way that they would be duplicated, which meant we would change the file name that was uploaded to a different one, and we would place it on a different folder than what Django wanted to do. It took me like one week to build the application,

and three weeks to build the logic to allow the application to save the files where we wanted them to, and it was really, really ugly code at that. Anyone else?

Hi, you said that Django and old PHP code coexist at the same time, right? You probably didn't just flip the switch, so can you explain this a bit more, like you probably couldn't use internal Django

road system and other features, like how did you manage to do this, that both systems coexist? Django and PHP share the same database with the same data. That's why we have some strange table names

and some older parts of the system. The URL routing was done both on the web server and the Django application, so some of the URLs would direct the PHP application, some would send you to the Django application.

Is that what you wanted? Okay, so after you launched fully Django, then you had to do some cleanup, the code. Yes, the code was... You basically hardcoded some of the URLs and some stuff. Yeah, as the PHP code was being removed, the rules that sent to it, the code itself,

it was deleted from the configuration. Any more questions? Sorry, it's not planned, but I can clarify the routing.

What we did is we had an NGINX front-end load balancer, and we had custom routes, a giant routing table in there that would selectively route traffic to the Django app or let it route back to the PHP app. We just slowly deleted those routes and let more and more routes fall through to the Django app. That's basically how we flipped over without doing a big bang release. So if things went wrong,

we could just simply add the route back in and went back to the PHP app again. Okay, thank you for that. So I hope to see you all at the lighting talks. Let's thank Ricardo. Thank you.