What about recommendation engines?
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Untertitel |
| |
| Serientitel | ||
| Anzahl der Teile | 118 | |
| Autor | ||
| Lizenz | CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben | |
| Identifikatoren | 10.5446/44778 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
| |
| Schlagwörter |
EuroPython 201993 / 118
2
7
8
15
20
30
33
36
39
40
45
49
51
55
57
58
63
64
66
68
69
71
74
77
78
80
82
96
98
105
107
108
110
113
115
00:00
PunktwolkeGoogolInklusion <Mathematik>RechenwerkKonvexe HülleMultiplikationsoperatorAlgorithmusVererbungshierarchieVorlesung/Konferenz
00:31
Gebäude <Mathematik>RechnernetzMaßerweiterungSoftwareentwicklerStatistische HypotheseGeometrische FrustrationInformationPrognoseverfahrenAlgorithmusKontextbezogenes SystemGeometrische FrustrationBitrateInformationMinkowski-MetrikYouTubeAbfrageComputerunterstützte ÜbersetzungGrundsätze ordnungsmäßiger DatenverarbeitungStatistikAdressraumDatenfeldWort <Informatik>Formale SpracheWeb SiteSchlüsselverwaltungKategorie <Mathematik>MultiplikationsoperatorDatenbankRechter WinkelQuadratzahlPhysikalisches SystemMaßerweiterungVideokonferenzBitStatistische HypotheseVorhersagbarkeitMinimalgradGruppenoperationProgrammfehlerSoftwareentwicklerWhiteboardComputeranimation
09:14
Mini-DiscFormation <Mathematik>Inhalt <Mathematik>DigitalfilterRankingPauli-PrinzipKollaborative FilterungSchwach besetzte MatrixBitrateApp <Programm>MathematikGrundsätze ordnungsmäßiger DatenverarbeitungAutorisierungCodeBitrateEntscheidungstheorieInternetworkingÄhnlichkeitsgeometrieYouTubeService providerGruppenoperationRechenbuchDeskriptive StatistikRankingSondierungKollaboration <Informatik>Inhalt <Mathematik>InformationEndliche ModelltheorieZahlenbereichRechter WinkelAlgorithmusSchwach besetzte MatrixCASE <Informatik>MultiplikationsoperatorRegistrierung <Bildverarbeitung>Physikalisches SystemGraphfärbungInteraktives FernsehenArithmetischer AusdruckDatensatzDateiformatSelbstrepräsentationHalbleiterspeicherProzess <Informatik>Formale SemantikKonfiguration <Informatik>DifferenteFilter <Stochastik>Computeranimation
17:42
MathematikPrognoseverfahrenMenütechnikBetragsflächePlot <Graphische Darstellung>StreuungPythagoreisches ZahlentripelAbstandEuklidischer RaumPhysikalisches SystemAlgorithmusPlotterÄhnlichkeitsgeometrieSystemaufrufDimensionsanalyseAbstandMathematikBitrateGraphPythagoreisches ZahlentripelRechenbuchAlgorithmische LerntheorieKoeffizientPunktDatenbankTreiber <Programm>Einfacher RingZeiger <Informatik>DreieckZahlenbereichWurzel <Mathematik>SymboltabelleQuadratzahlPotenz <Mathematik>KontrollstrukturMehrrechnersystemAusdruck <Logik>Computeranimation
22:51
ÄhnlichkeitsgeometrieFunktion <Mathematik>PrognoseverfahrenKollaborative FilterungMittelwertTreiber <Programm>DatenbankKontinuumshypotheseRankingCOMBildschirmsymbolDatenfeldTreiber <Programm>FunktionalBitrateVorhersagbarkeitAbstandAlgorithmusMultipliziererFunktion <Mathematik>CodeÄhnlichkeitsgeometrieSichtenkonzeptSchwellwertverfahrenRechenbuchEuklidischer RaumMittelwertRankingAuflösung <Mathematik>XML
27:54
CodeRückkopplungZeitrichtungAlgorithmusRückkopplungCode
28:24
HypermediaVorhersagbarkeitProjektive EbeneWeb SiteProdukt <Mathematik>Geschlecht <Mathematik>E-MailFormation <Mathematik>Logischer SchlussVorlesung/Konferenz
Transkript: Englisch(automatisch erzeugt)
00:05
Hello, everyone. It's super nice to have you here today. This is the first time I talk to this amount of people. So, I'm kind of nervous and anxious. Thank you for being there for me. This is an amazing experience. Thank you very much. So, today I'm going
00:27
to talk about recommendation engines and how to build one simple recommendation algorithm. But first, if you don't mind, I would like to give you some context about me and also
00:44
break the ice. So, I come from Brazil with the S, actually. I know that you know Brazil with the Z, but we write it with the S. We have more than 8,000 square kilometers of
01:01
extension, and it borders like 10 countries. This is huge. But I come from the very south of Brazil. I will show you. This is in the Rio Grande do Sul. So, a little bit
01:22
about Brazil. I don't know if I have another Brazilian here. Oh, nice. Nice to see you. Portuguese have landed a bit more than 500 years ago. We have 30 years of a weak
01:44
democracy, and I don't know if you heard, but we kind of are having coup d'etats and coup d'etats all the time. We have unlimited natural wealth. And how are you?
02:01
Oi, todo bem. Well, so from the very south of Brazil, it's Rio Grande do Sul. We like Churrasco and Chimaro. We don't have much of Somba, neither the warm weather, actually. We board Argentina and Uruguay. We are full of playing fields, so we use
02:25
it very much for agriculture and cattle breeding. Our daily temperature can vary from 28 to 10 degrees in the same day. And the most traditional thing we have in Rio Grande do Sul is Ervamati tea. It's called Chimaro. So, this thing in the
02:48
lady's hand is the Chimaro. You can see a very huge piece of meat that is being cooked by the fire that is under the ground. So, I don't know. I
03:09
don't know why. Maybe it's because it has caffeine. Some footballers have been drinking it, like the England's footballers. And Messi, he's Argentinian,
03:25
right? You know that Brazil's Argentinians have some struggles. And this is Renaudinho, also drinking the Chimaro. He's from Porto Alegre as well. And the curious thing is, it's a hot beverage. And we drink it on the beach in the hot
03:44
weather. It doesn't matter. We like it. So, this is me. I'm a developer. I'm an economist. I'm doing a master's degree in statistics. So, I'm a date enthusiast. I'm a cat person. And I am addicted to travel. This was me, my first
04:07
travel, talking about how to deal with the frustration of not having your hypothesis proven as a data scientist. I don't know if I have data
04:20
science here, but sometimes we spend like days trying to prove a hypothesis. And it doesn't happen like we want. But the thing is that even if it's not a good hypothesis, I mean, it's not accepted, we can gather information of
04:40
this. So, yeah, this was in the human data science. Okay, let's go. What about recommendation systems? So, the key word for recommendation systems are revenue and customer engagement. What happened is that we as customers are overloaded
05:07
with information. So, we have a lot of items out there, a lot of movies to watch, a lot of, I don't know, videos, and we don't know what to watch sometimes or we don't know what to buy sometimes, what is the best for us.
05:24
So, we have a lot of information. This is a very new personalized way of selling, buying, watching, and getting to know things. And, well, McNee said that it helps groups of users to select items from a crowded item or
05:50
information space. Well, Amazon, YouTube, Netflix. I mean, there is a lot of
06:01
others like Udemy, Google, oh my God, almost every website, every e-commerce uses it. But, just to bring some examples, where Amazon, in a quarter, they had almost $13 billion of revenue. And, this was the first quarter they
06:29
implemented the recommendation engine. And, they had $30 billion and it was 30% more than the same quarter of the last year. So, for YouTube users and for
06:46
YouTube, more than 70% of the user consumption come from recommendations. So, when you were watching a video and there's another video coming or you go
07:00
through the feed and there's some videos recommended for you, almost 10% of the consumption of YouTube is about this. And, for Netflix, 75% of the consumption comes from recommendation. So, we can see it's a very big matter
07:22
recommendations. So, when we need to recommend something, we might be looking for answer two problems, prediction and rating. There is an approach for recommendation that is being used that has not artificial intelligence that is
07:49
like, oh, the most sold, the most clicked items that it appears and it can be easily made with a query in the database and showing the user what items
08:03
have been more sold and everything. But, what we want to have is things that the customers are likely to love, are likely to buy, are likely to watch.
08:21
So, we don't need just a predicted rating for an item, but either if they might like it or not. Okay, a little bug there. But, how do you get this data, right? So, data is the key. And, we have two kinds of data. I'm saying two
08:47
kinds because I'm separated that way. I don't know, maybe we have more, even more than we haven't talked about, but we can put everything in these two categories. So, implicit data. It's about tracing. It's about the data that the
09:05
customer didn't give to us, like, their name or address. But, where did they click? What kind of movie they like? And something like this. We can collect
09:20
this data, right? Big data. And, we have the explicit data. This is the more difficult to get because it depends on the action of the user. So, answering a survey or rating items. I don't know. I think I have never rated an item. I don't know about you. But, it's not common. I mean, thank for
09:47
the people who rates because I'm always going to the rating. But, yeah, I have never. So, this is the basic models of recommendation systems. We
10:03
have AI, okay? So, we have the content-based, the collaborative, and the hybrid solutions. I mean, these companies like Amazon, Netflix, YouTube, they are using more the hybrid solutions that is very based on the
10:24
content, but it's very based on the collaborative and has some secrets on the recipe that we don't know. So, content-based. The recommendations are based on the description of the item or in the synopsis or in the
10:43
genre or even in the author. There is an article in the internet showing that an author had launched a book, like, in 2010, and the book was not
11:04
very good sold. That's okay. In 2015, another author launched another book, but the content was very similar to this one that has been launched in 2010. And what happened was that the book that was launched
11:26
in 2010 started to have a lot of people buying it, and the author was like, oh, what's happening? Then, tracing it back, they have seen that
11:41
because of this book that was very similar to the 2010 book, it starts to sell more. So, it happens. So, content-based recommender systems are born from the idea of using the content of each item for recommending
12:05
purpose. It avoids the cold start recommendation problem. Wait for it. I'm going to talk about it. And content representations are open up to options to be used with different approaches, like, you know, or text
12:26
processing techniques, semantic information. It has a whole world of tools to analyze this data. So, another one is the collaborative,
12:44
and it's memory-based. Recommendations, so, in this case, are based on user social interaction and rankings provided by other users. This collaborative model
13:01
is called collaborative filtering, and it's divided by user-based and item-based. So, let's see this problem. It's Saturday night. I am at home. I open my favorite streaming app, and I don't know what to watch. I
13:33
don't know what to watch. Maybe it's a good idea to ask her for a movie recommendation. And this is the truth, okay? Marta is my friend. I
13:44
didn't know what to watch. And I was, oh, my God, I'm always like this. I don't know if you believe it, believe in this kind of thing, but I'm Demini, and I can make decisions. So, yeah, I was like, oh, my God, what am I going to watch? So, I'm showing you the user-based
14:02
filtering. This on top is my friend Henry. He likes orange, grapes, raspberry, and banana. Marta likes grapes, and Demini likes grapes and banana. Maybe Marta could like orange, or banana, or raspberry,
14:25
and she never have tasted it. Maybe I can recommend her. But looking at this picture, I can see that Demini is more similar to Henry,
14:41
because she likes more fruits. Maybe Marta just like grapes. And if she likes more fruits, and she's more similar to Henry, maybe I can recommend her oranges and raspberry. Well, but what about the item-based filtering?
15:02
The thing is, again, Henry likes the same fruits, Marta as well, and Demini as well. But if I go and check the items, I see that more people like banana. Maybe I can recommend Marta a banana, and maybe
15:26
she likes it. So, it has been shown in the market, the collaborative filtering, more accurate than the content-based. It's easy to
15:46
implement, but sometimes it's not a good idea to implement. We have to keep this in mind. When is not a good idea? When we don't have enough knowledge about the item and the user, so we don't have enough ratings, I don't know nothing about the user, I can't do
16:05
the math to see with what user or item this user is more similar. So, how can I recommend something? And when the item has not been ranked enough. So, it's a new item, and I don't have any
16:23
information of this. So, a problem. Code start. Code start, as I have said before, and now I will explain, is the expression we use when we don't have much information about the user and
16:40
would like to recommend them something. I think that helps on the code start is, I don't know if you have Netflix, or there are other sites like this, like Pinterest, you go for the first time there, you do your registration, and the website asks you to select items that you like more, or
17:05
gems that you like more. So, this helps the algorithm to recommend you things and avoid the code start. So, it's sparsity, that is the problem when we don't have much
17:22
number of ratings. And new items, because we don't have information. So, it's not a good idea when we don't have any information, because this collaborative filtering is very based on similarity calculations. So, let's build
17:45
our own recommand system algorithm. These are the steps, the basic steps to build the recommendation system. So, first, we need to call, we might
18:03
choose how to do the math to find the similarity coefficient between users. Then, we have to predict. So, we have to find the predicted score for the movies that the user didn't watch, and tell. So, let the user know what you have predicted to them.
18:24
So, recommend it. So, let's go back to my problem. It's Saturday night. I want to watch a movie. I don't know what to do. So, maybe I will ask Martha, but may I try another friend?
18:43
The thing is, I have this database, and as I said, it's true. My friends gave me these ratings. It's good, because now we know how similar we are, and how to get recommendations. So, this is our database. Let me see if I can have a pointer here.
19:04
Okay, so, the Wolf of Wall Street. Danny hasn't watched it. Cool Runnings, I didn't. Baby Driver neither. Danny, as well, didn't watch the Cool Runners, and we didn't watch
19:21
Baby Driver, and Ulta didn't watch the Lord of the Rings. So, let's do the call to see the similarity between us. What I have done here is I have plot a two-dimension graph
19:44
of the Wolf of Wall Street and The Devil Wears Prada, and I'm seeing how the data is dispersed. So, here, I can see that Martha gave a three rating for The Devil Wears Prada
20:03
and three for The Wolf of Wall Street. I gave four, and four, and Philip gave three for The Devil Wears Prada, and four, and fifty for The Wolf of Wall Street.
20:23
When I see this graph, and I try to understand who is closer to me, because I'm trying to get the similarity here. So, when we talk about data similarity, it's much about where are we together, what data points we share, and things like this.
20:50
So, when I want to recommend something to anyone, I really want to recommend to someone that is very similar to me, so I know that this person is going to like it.
21:04
So, thinking about it, I can try to measure the distance. So, from Philip, I'm 0.5 distance, and from Ulta, well, this is a triangle. So, let's do the hypotenuse,
21:27
doing the Pythagorean. So, do you remember that from school? So, yeah, we get the distance of this cathedral, and this one, then we can doing the square of those, summing up,
21:46
and the square root, we get the hypotenuse. So, doing this math, I know that I'm 0.721 distance from Ulta. So, I'm more distant to Ulta than to Philip.
22:02
This is the Euclidean distance. This is also a math that is used for the k-nearest neighbor. This is a very used algorithm in machine learning. I would just show this formula, but I thought that if I showed you the triangle and the Pythagoras, it will be easier, because
22:27
I don't think people like symbols, characters, and numbers, exponential, everything together. So, this is just the Pythagoras with summing up everything with all dimensions we have,
22:41
because I have showed you the two-dimensional graph, but if I have a lot of movies, I will have a lot of dimensions. So, this is how we are going to make the similarity calculation.
23:03
If I do function named similarity, get similar, sorry, I did this. I will show you the code, and it shows that regarding all movies, I'm more similar to Ulta than to everyone else.
23:28
So, let's predict. The prediction is that I have to predict what is going to be my rating for Cool Runnings and for Brave Driver, because maybe
23:44
that's the movie I'm going to see tonight, actually on Saturday. So, here we can see that we don't have much ratings. I mean, like, four of those,
24:01
more plus mine are missing. So, what if, like, Octavio, that is the most similar to me, like we see here, hadn't rated movies. Let's suppose that we have, like, 50 movies, not just
24:22
five. What if a person appreciates a movie that everyone else has low rated it? A solution to this problem is to use the weighted average. So, let's go to the Saturday night. What movie should I watch? Pro tip. It's up to us to choose what is the recommendation threshold.
24:49
We have this method, the method that is to get my average rating and just recommend if the prediction is higher than my average rating. So, recommendations for me. Here is the
25:12
code I have done for the recommendation. This measures the Euclidean distance
25:21
and multiplies for the similarity for the weighted average. So, here, what happened is I have put the similarity here. Sorry. The predictions we have, Danny hasn't rated Cool
25:42
Runnings, so it's blank. And I have multiplied this two to get the weighted rank. And in the end, we get a prediction for the Cool Runnings. For example, for me,
26:05
I would rate Cool Runnings as 3.6. At least the algorithm says that. And the same thing for BraveDriver, I would rate it as 3.93. So,
26:26
here's the let me see here. Yeah. This is the function of the recommendation. And this is the output. So, as we have seen, I would recommend more BraveDriver than Cool Runnings. What does it mean? That I may be recommended with BraveDriver by the algorithm.
26:48
So, I should tell my users what is the predicted. Here, doing the prediction for everyone, I have seen that I would be recommending Cool Runnings like this, BraveDriver like this,
27:10
for Danny. These are the predictions. And at the Saturday night, I'm going to see BraveDriver.
27:22
So, problem solved. As I have seen here, my average rating is 3. So, these two can be recommended. Maybe in my Netflix or my streaming app, these two are going to appear
27:42
like watch it now. Cool Runnings and BraveDriver, they are likely to be successful of my good movies. So, the code is here. If you want to check if you want to see and see how to
28:02
make this algorithm. So, tiny.cc, 2019. And thank you very much. I would love to hear a feedback from you. And I'm here if you have any questions. Thank you. Maybe one question here. Thanks for the talk. I'm curious, at your company,
28:48
what products or items are you recommending? Nice. So, I'm a data enthusiast. My company is ThoughtWorks. We don't have much work on recommendation. We have been doing some inferences
29:08
like we have made for a big media company gender prediction. So, when a user enters in the website,
29:20
is they a male or is a male what age? And kind of this. But recommendation, we haven't done it yet in a project. Thanks. Welcome.