Relevance is not a Thing but a Perception
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 56 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/67179 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Musical ensembleFront and back endsMereologySoftware developerDemo (music)Client (computing)XMLUMLLecture/Conference
00:28
Object (grammar)Real numberElasticity (physics)Linear regressionRandom numberOrder (biology)Library catalogVacuumData dictionaryDegree (graph theory)Different (Kate Ryan album)Machine visionOperator (mathematics)Product (business)Type theoryInformationPerfect groupMultiplication signTouch typingWordComputer fontSoftwareINTEGRALHookingMoment (mathematics)ECosSearch engine (computing)Inheritance (object-oriented programming)Tape driveModal logicSmoothingPopulation densityData storage deviceImage resolutionVideo gamePoint (geometry)Suite (music)Hill differential equationSineGame theoryNP-hardHelmholtz decompositionWell-formed formulaKeyboard shortcutElectronic mailing listAstrophysicsOvalBlogMereologyResultantPhysical systemComputer programmingAlgorithmFrame problemPresentation of a groupSeries (mathematics)Equaliser (mathematics)Atomic nucleusOrder (biology)Spring (hydrology)Data conversionAverageWebsiteRing (mathematics)Total S.A.Bit rateExpressionSlide ruleIntegrated development environmentContext awarenessProfil (magazine)Special unitary groupProcess (computing)1 (number)Shape (magazine)Internet forumPosition operatorLevel (video gaming)Prime idealInsertion lossDescriptive statisticsData acquisitionLibrary catalogGroup actionImpulse responseSelectivity (electronic)CountingSpeech synthesisFluid staticsFacebookHand fanRevision controlTwitterWikiFreewareGoodness of fitSet (mathematics)Flow separationFormal languageDevice driverProjective planeWater vaporSystem callBlock (periodic table)Wave packetMatching (graph theory)Computer fileReal numberConnectivity (graph theory)Electronic program guidePhase transitionWeb pageNormal (geometry)Query languageObject (grammar)BitWeightField (computer science)DemosceneSocial classRandomizationTask (computing)Row (database)Computer animationLecture/Conference
09:55
Ext functorState transition systemGenderStreaming mediaCategory of beingSquare numberLogic gateCommon Information Model (computing)Right angleHookingResultantPressureInformationFree productProduct (business)Game theoryVideo gameGoodness of fitSet (mathematics)Row (database)MereologyPower (physics)Formal languageOffice suiteBoiling pointWeb crawlerHill differential equationSoftware testingCommodore VIC-20Focus (optics)40 (number)Series (mathematics)Bridging (networking)Connectivity (graph theory)Software developerProjective planePoint cloudImage registrationMathematicsHand fanBoss CorporationWordAnalytic continuationQuery languageContext awarenessMaxima and minimaElectronic mailing listSimilarity (geometry)Barrelled spaceOnline helpWater vaporInheritance (object-oriented programming)Multiplication signBus (computing)Electric generatorRevision controlPosition operatorComputer fileConfiguration spaceImmersion (album)Graph coloringDynamical systemDifferent (Kate Ryan album)Flow separationWorkstation <Musikinstrument>Medical imagingSelectivity (electronic)Event horizonWaveCellular automatonFluid staticsOcean currentPhysical lawOrder (biology)Musical ensembleFilm editingSymbol tableGroup actionDemo (music)Insertion lossBlogPlastikkarteBit rateSoftwareMagnetic stripe cardWebsiteLink (knot theory)Process (computing)Web pageFunction (mathematics)Theory of relativityTask (computing)GenderType theoryWritingField (computer science)Sampling (statistics)Attribute grammarArithmetic meanElectronic program guideComputer animationLecture/Conference
19:04
Category of beingQuery languageConfiguration spaceView (database)Staff (military)Intrusion detection systemConnectivity (graph theory)Information privacyFilter <Stochastik>Set (mathematics)Line (geometry)Mobile WebType theoryBlogSoftware testingMultiplication signError messageCategory of beingLink (knot theory)MathematicsComputer animationLecture/Conference
20:40
Demo (music)Plane (geometry)Computer animationLecture/Conference
21:02
Multiplication signRankingWater vaporContext awarenessCartesian coordinate systemPhysical lawMetropolitan area networkSolitary confinementEndliche ModelltheorieStandard deviationNumberLecture/Conference
21:50
Product (business)Software developerRight angleNatural languageScripting languageConnectivity (graph theory)Level (video gaming)Food energyTask (computing)Square numberRobotFlow separationLecture/ConferenceMeeting/Interview
22:31
YouTubeEndliche ModelltheorieMoving averageQuery languageLecture/Conference
22:59
Category of beingInformationLocal ringEmailData storage deviceNavigationElement (mathematics)Image resolutionProjective planeHill differential equationResultantGame controllerQuery languageComputer animationLecture/Conference
23:45
Einstein field equationsTouchscreenDemo (music)BitSpacetimePerspective (visual)Query languageExpressionState of matterMusical ensembleLecture/ConferenceMeeting/Interview
24:26
BitMereologyDebuggerVideo gameQuery languageMusical ensembleMobile WebConnectivity (graph theory)Lecture/ConferenceMeeting/Interview
25:07
Query languageMeasurementElectronic mailing listType theoryCountingConnected spaceHeat transferEvent horizonState of matterOrder (biology)AuthorizationLecture/Conference
25:46
NP-hardMultiplication signRoundness (object)Musical ensembleLecture/ConferenceJSONXMLUML
Transcript: English(auto-generated)
00:07
OK, so good morning. My name is Ana, and I'm a backend developer at Empathy, who usually works at creating the backend part of demos for new clients. And today, I want to talk about relevance, how I think that is a perception and not just
00:22
a thing, not just a score, and how we work with it at the company. But first of all, let's define what is relevance. I've been looking for different definitions, and I really like this one from Cambridge Dictionary. It says that relevance is the degree to which something is related or useful to what is happening
00:41
or being talked about. This is a really wide concept, and it makes me realize that we work with relevance in a lot of moments of our life, like when we are at the school and we have to study for an exam and we have to remark the most relevant parts of the summary so we don't have a good grade, or when we are at our house and we have to do the chores
01:02
and we don't have time to do all of the chores so we have to do the relevant ones first so we can keep our house decent, or every day at work when we have to prioritize the task and we can develop the project. So if we have to find relevance in a lot of moments of our day, it should be easy for us to heed
01:22
what the consumer wants to find when he's using our search engine. No? Well, in real life, we can see that it's not that easy, and there is not just a unique solution that will fulfill all of our consumer necessities. And I have the perfect example for that. So here, we can see a willing teapot,
01:43
and our protagonist is going to keep it away from fire, but the telephone starts to ring, and his father tells him that that is the relevant task. But this task is so immediate that Pingu doesn't know what to do, and he starts crying. But his mother checks the teapot and checks the phone, and also tries to cheer up his person
02:02
that is also a relevant task. I know that this can be just a funny scene from a kid show, but it reflects reality. Relevance is not easy to estimate. It's not trivial. That's why we all have had that one classmate that has a fluorescent book. And if you didn't have that one classmate,
02:20
it probably means that that class means what's you. That's why sometimes we prefer not to have any guests at our homes because our house is far from being decent. And that's why sometimes our projects fail because we work with relevance in a lot of moments of our life, but it remains as an absent consent.
02:43
Let's watch the father. Relevance is subjective. We have Pingu who didn't know what to do and didn't have time to think about it. We have Pingu's father who thought that the call was the relevant one because it could be an important call. And we have Pingu's mother who thought that the tip was the relevant one
03:01
because you could finish it quicker. It's objective. It depends of each of the characters of the situation has a different way to find relevance. And that's okay because that's what happens in real life when our consumer clicks on a certain thing and tries to find his relevant product.
03:20
It's objective. And what is subjectivity? A perception. So if relevance is, if subjectivity is a perception and relevance is objective, it leads us to the title of this talk, relevance is not a thing, but a perception. So now that I convinced you about the main point
03:42
of this talk, let's start talking about search. Relevance in search is really important because when a consumer enters our website and clicks on a certain thing and tries to look for a product he expects to find the product he's looking for in the first position of the result list
04:00
because of the position bias. So he's not likely to scroll too much or to go to the second page. And we don't have three situations. First one is the perfect one. The consumer finds the product and he will return in the future. He buys it. The second one is that the consumer don't find the product but he finds something that he likes,
04:20
something related to it. So he won't buy the product but he will return in the future. And the last one, the worst scenario possible. The consumer doesn't like what he sees, doesn't find the product and will never come back. So I think that we can probably think that the best solution is to find the perfect algorithm for relevance, the perfect formula
04:42
so we can always show the consumer the product he wants. Well, when I started researching for this talk, one of my workmates recommend me this talk by Roman Gravenikov in my since 2019 where he talk about personalizing search results and the position bias and everything.
05:02
And he does this really interesting experiment where he takes a baseline algorithm from Elasticsearch and change the algorithm so he can improve the conversion and the average order value. I bring this experiment here because of this last row because when he uses a total random algorithm,
05:21
of course it's worse than the baseline but it's not that much. In fact, when we see the algorithms that have better performance, yeah, they are better but not that big difference. So I think that we can conclude that even if there's a way to improve our relevance
05:41
with a formula, it's not a big difference and will probably cost us a lot of time to find the perfect solution. So returning to the main point, if we said that relevance is a perception, a formula won't meet all of our needs because we need to look not only on the result list
06:00
but also on the whole shopper experience because if the shopper is the only one who knows what he's looking for, let's listen to him, let's try to understand him by the things he does to our website, the actions, and let's not impose any result to him. And now I want to talk a little bit
06:21
about Empathy's approach. So when we receive a query on our search API, we dynamically generate a match that we want to the catalog and return a result. In this match, we can see this tool called Equalize that we use. We give it this tool to the merchandiser
06:42
so he can have some static relevance to the search because we allow him to select some fields that he knows that are more relevant, that he wants us to take more into account. So they can have more weight to their final result
07:03
like selecting brand over description. So if a consumer search for cola, he will probably see first Coca-Cola products than a normal cola product. And we also have Contextualize, but Contextualize is a tool that deserves his own life. So we are going to talk about it next.
07:22
Now what I'm going to do is that I'm going to take you into a journey because in Empathy, we believe that the experience of searching is an inspirational process and there are some faces and there are some user feelings that we have to take into account. So let me take you into a journey for some components that are really interesting.
07:43
First of all, we have the Click component. It's the first component that the user, the consumer, is going to see when he enters our site. It's a component that, by the name, you can see that will show the most clicked product. But it will show these products in a face that we call pre-searching phrase.
08:01
It's a face where the user is open to discover, is open to be inspired. It's just like when you are on Instagram and you want to find a tag or a publication and you enter and you, of course, can go to the search part, but you first is going to see the wall, the publications.
08:22
You are going to see what is happening right now and you are being inspired by what is happening right now. It's the same. It's the way we are presenting us to the consumer and it could happen that the consumer finds the product he's looking for without typing anything. It will be a perfect scenario. But even if he don't find any product,
08:41
we are telling him, hey, we have this. This is what is being popular and this is how we are presenting us. It's a good starting point to meet the consumer. Next, next is the consumer clicks on the search bar. He is going to find the Empathize component.
09:01
Empathize component is a component that have two different functionalities, two different aims. Depending on how the consumer interacts with it, it could happen that the consumer just click on it and doesn't type anything. It means that we are still on the pre-searching phase. User still wants to be inspired. So we are going to inspire him with some popular searches
09:21
that come from anonymized data from other shoppers. So he can know what is happening right now and he can also find a product without typing anything. That is also a good starting point. And if the consumer starts typing something, Empathize will change and will take into account what he's typing, the letters, the words,
09:41
and will start sending some suggestions because right now, when he starts writing, he's entering the searching phase and he doesn't want to inspire. He wants to be guide to the product he's looking for. So if I'm the consumer and I'm looking for a cotton shirt, but I'm writing just shirt, Empathize will start showing me some black shirt,
10:02
a striped shirt, cotton shirt. So I can click on some cotton shirt and see more relevant products with that query that with the shirt one. I'm being guide to the product I want. And next, when we see the results, we have Contestalize. That is the tool that we saw earlier.
10:22
Contestalize is a tool that will take some events, some events from other shoppers, like click or artichoke, and will take them into account to reorder the result using the wisdom of the crowd. So if I'm looking for a cotton shirt and there are a lot of events on white shirts, I'm going to first see the white cotton shirts
10:43
on my query because that is what's being popular right now. That is what people around me is looking for. And we let the merchandiser select which field is going to be taken into account and my sample is the color of the shirt. But it could be the brand, it could be the size, it could be whatever the merchandiser wants
11:00
because he can personalize it whatever he wants. And this is being dynamically generated so it will evolve with time. It will evolve. Well, today, a white shirt is being popular but in the next week is blue shirt so it will change and the merchandising doesn't need to do anything.
11:21
Next, in the results, we can see some related tags. Related tags is a tool that will help the consumer to reduce the noise, to refine the query, to refine the results because it can be seen as a filter because there are some words there
11:41
that will help us to refine the query. Like I can write cotton shirt and have white strip and black and anything, a word. And it could help us to reduce the products. It could be seen as a filter but it's not because this is being generated by another queries of the suppressed anonymized data.
12:04
And it's more concrete. This is not like a general filter. This also evolves with time and it can be used with a filter at the same time and we can do one, two, three related tags, more than one filter. It's the way we help the consumer.
12:21
We guide him to the product he's looking for and so the product can appear on the first position of the result list. The consumer is generating his own relevance. And now the consumer has the product he wants. He found it thanks to contextual, related tags, filter, whatever. And when he finds the product,
12:42
he changed his intention of searching because he already has what he wanted. So he probably is looking for something new. We're going to help him find this new product with next queries component. Because next queries is a component that will help, will show the consumer the next popular queries,
13:01
that's another super search before that one. So if I'm looking for cotton shirt, this I can look for pants or for shoes. Without typing anything I can see that and I can be also inspired by that new intention. We can help him to find his new intention with next queries.
13:22
And what he's going to find, what he's going to find when he starts searching again? He's going to find again the related tags and the contextualize but he's also going to see promotion and banners. That is some, this can be seeing a static promotion with everything but we let them
13:41
and I see personalized theme, personalized this. And select which queries he wants to put a promotion there. Select different promotion depending on the query. And it uses the marketing bias or promotion bias. There is a principle that said that when you promote a product,
14:00
it will be more click, it will have more relevance. So we can use this as well. And with this, I want to tell you that we let the merchandise override our configurations because we listen to the consumer but we also have to listen to the merchandiser that is the one who knows what he wants to do his job.
14:23
So we let him override some things. And I want to show you two more components for the merchandiser that are really interesting. We have boost and beauties. That is a tool that we allow the merchandiser to select a query, a certain product, and put them first in the result list,
14:42
put them last in the result list, play with it a little. And he can also select some products that share an attribute like boost all the blue shirts, boost all the white shirts, beauty them, whatever. He can personalize the queries he want and he can create his own journey.
15:02
And also we allow them to use Synonymize. That is a tool that will let the merchandising select some query that has a similar meaning to another one, another certain, and allow the search to show the same result list
15:21
as in the original one. It could be really helpful if you have a query that has no results and you are seeing that a lot of consumer are looking for the query and this is similar to another one. So you can help the consumer not to stop abruptly his journey and continue. And also if you have a query that doesn't have a lot of relevant results,
15:42
you can also use Synonymize to personalize it and change the journey. And now when I started this talk, I said that I worked as a backend developer for creating demos. So I think that you probably thought that I would bring a demo here,
16:01
but no, I didn't bring one demo, I'm sorry. I bring two. So we're going to see some of these components in action. First of all, we are going to see a desktop one. We are going to click to start. We are going to look for some fashion. When we start, we are seeing here the top clean
16:21
because right now I'm the consumer and I want to be inspired by these clothes. I want to see the historic queries that I can just clear because I've been doing some tests before this talk. And I can see also the popular searches here in empathize because when I enter this site, I already have the focus on the search bar.
16:42
So I can just go here, be inspired, look for something, and then I'm going to start typing. I'm going to look for a shirt. And now empathize is changing what he's looking for right now. He thinks that I'm looking for a shoe. I want to look for some hints shirt. So let's look for shirts.
17:00
And now we can see a promotion that the merchandiser that we put there because we are the merchandiser right now in this demo. And we can also see this. This is really interesting because we can select how many products we want to see in each row. I didn't talk about this in the talk, but it's really important how many products we put on each row because if I put six,
17:23
it is really probably that I as the consumer will see this part of the products because there are a lot of information right now. So it's better to go like four or three rows so we can see everything. And here we can see the related tasks. We can look for some white shirt and everything is going to change.
17:40
It's not like a filter because it's more concrete. And I can also filter by gender like some main shirts for my boyfriend. And now we can sort also, we can clear. And if I click here again, I'm going to see the next queries. So probably I want some sweater. This is also really important, okay,
18:01
to write, to help the consumer not to end his journey if he doesn't find any results. So if I write anything, I'm going to see that there are no results, but I'm still seeing top click. I'm still seeing products. My journey is not ended. I just need more inspiration to look for my product.
18:21
And also we can not only show products, we can show anything more like a link to our site or a link to everything. So here you can see the empathy page if you want to visit it and find more information. And also I'm going to show you some synonyms.
18:41
We can look for a skirt. We can see these marvelous skirts. And I can look for mini. And I want to see the same skirts. That one, that first skirt has moved back down. That's because me as a merchandise, I decided that in this query that is being synonymized with a skirt,
19:04
I'm going to take the shirt and view it. Because even if they are synonyms, I can personalize them. They are not a copy. They can be personalized like they are a normal query. So I can play a lot with these configurations. And now I'm going to show you the mobile one.
19:22
I know that it's a little bit small, but I think that we can see it right. I'm just going to click here and we can see here the history that I'm going to clear. Some IDs because we can search by ID, but it's another component that I'm going to tell today. I'm sorry. We can see what is trending right now that this is empathize.
19:41
And we can see the top click. And also we can see here some privacy concerns because you know that we have empathy. We try to think about privacy in our users. I won't talk too much about this, but I can see here that I can personalize. I can tell him to save my search history, suggestions, my style and everything.
20:05
I don't want to save this experience. I want to search for a shirt. And we can see empathize moving and trying to help me to find the shirt that I'm looking for, to find some black shirt, to find some shortest leaf shirt.
20:20
And everything. And we have here some search and filters. These are the same components. And now we can see the next queries. People also search for. So we can play with these components. We can create our experience as a merchandising, whatever we want. We can respect our consumer's data.
20:43
And that will be all. So thank you for listening to me. Thank you.
21:02
Thank you, Maria. We have time for questions. Who'd like to start? Hi, great talk. Just a couple of questions actually. So like how many, so in your contextualized step,
21:23
how many days of data do you usually use? And like what model do you basically use for building those rankings? In your contextualized step, you're using historical data to basically build those rankings, right? So I'm asking how many days of data
21:42
do you have a standard number of days that you use? And what kind of model do you use behind those ranking? Well, for the, I don't know. If I understand you right, I create demos. So all of the data that I use is generated by an internal bot. But in real product, we use the data for several days.
22:02
I don't know if there are three, four, five days. They are continuous evolving. And these are all data generated with the clicks and with everything, with anonymized data, with just the session ID. And that's it. We just take this data, we generate it for each of these batches, written tasks and squares that are different.
22:22
And the magic starts because it's not my team, right? They create magic and now I have data and I have to do these amazing components. Thank you. Anyone else? So you mentioned that as it's anonymous data, right?
22:46
Another question is, but I was also seeing you have those historical queries being shown. How do you get those historical queries if you're not tagging users?
23:03
Okay, so I'm going to try to show it again, the history queries. So if I click here, here. You can see that these are just the queries that I did, even though no results won, they have no sense. But all of this information is being stored right now in my local data in the navigator, yes.
23:23
So I can just click here or I can go to inspect element and delete all the local storage because it's been here in our right. So we choose the data, but I have the control right now. And I know that if this is a real project, I will just delete it and it will be free. And nobody knows what I did right now
23:41
and everything will be that easy. Thank you. Yeah, first, thanks for the talk. I think that was one of my favorite talk in the conference so far. So I have two questions that are somewhat related.
24:01
So the first one is, so you showed a lot of demo example based on web. And one of the challenge, especially on the UX side, when you're on mobile app, you have much less screen estate. And so I want to hear a bit more about your perspective about like things that are like query suggestion, tag suggestion, et cetera, like how do we evaluate
24:22
what works and what doesn't on mobile because you have much less space to try for? Like how do you measure whether it's actually relevant for the user or not? And if you could talk a bit about that. Well, I work as a backend. So this is more like a front end part. I can really tell you how many queries could it be.
24:41
But I know that probably when I talk with front end, we say that there are like three, four, five, it can change even in the device, they can change it. So more or less. And when you use these companies, they will first show you the most relevant ones, even in the queries. So even if you just saw three,
25:00
they are going to be the three more relevant. So for mobile, it's okay to show just that. How do you measure it's relevant for the user or not? How do you measure it's relevant for, how do you know it's relevant to the user? We know that it's relevant because there are a lot of queries made, there are a lot of queries that,
25:21
there are a lot of people that search for meaning right now. There are a lot of people that search for skills. So we just count the queries and we just see if they have clicks on there, if they have a two car events, we just order like that. With that event, that anonymized data, we connect it and I don't know, I just have a can, I don't know about data.
25:41
I can tell you later if you want. One more? We have time, so if you have more questions, please don't be shy. So then I think we are ready. Let's give a round of applause.
26:01
Thank you.