Behind the Scenes of an Ads Prediction System
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 131 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/69407 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2024127 / 131
1
10
12
13
16
19
22
33
48
51
54
56
70
71
84
92
93
95
99
107
111
117
123
00:00
DemoscenePredictionSystem programmingTrigonometric functionsNetwork operating systemMathematicsMachine learningComputerRadiology information systemOperations support systemMetric systemMetropolitan area networkVirtual machineComputing platformDataflowAlgorithmRight angleData conversionSuite (music)Moment (mathematics)ECosProduct (business)System programmingAutomatic differentiationSelf-organizationDigital photographyResultantAreaQuantumPrediction1 (number)View (database)Electronic mailing listComputer configurationQuicksortKey (cryptography)Theory of relativityMessage passingMusical ensembleQuery languageSet (mathematics)Point (geometry)Repository (publishing)Tablet computerFacebookLenovo GroupComputer animationSource codeLecture/Conference
07:38
2 (number)Bit error rateMetric systemComponent-based software engineeringEndliche ModelltheoriePredictionData conversionWave packetAutomatic differentiationReal-time operating systemNear-ringSource codeComputing platformFront and back endsInformation privacyMultiplication signRight angleProcess (computing)Key (cryptography)System programmingStructural loadProduct (business)GoogolComputer animation
10:23
Data conversionBit rateRankingOrder (biology)Information securityDigital signalMobile appMathematical optimizationFigurate numberProjective planeData conversionAutomatic differentiationMereologyKey (cryptography)QuicksortOnline helpRight angleBlogProduct (business)Computer animation
11:49
Design of experimentsInternet forumInformation securityDigital signalAutomatic differentiationData conversionRight angleSet (mathematics)Computing platformProfil (magazine)GenderSoftwareFormal languageMathematics
13:12
Price indexRankingSocial softwareView (database)Automatic differentiationWave packetKey (cryptography)Medical imagingData typeCondensationGraph (mathematics)Right angleInformationVideoconferencingLatent heatEndliche ModelltheorieMultiplication signFile formatComputing platformComplex (psychology)YouTubeNumberLevel (video gaming)StapeldateiLaptopComa BerenicesSystem programmingForm (programming)Phase transitionQuery languageContext awarenessDivisorMereologyComputer configurationSelectivity (electronic)Limit (category theory)Sampling (statistics)HypermediaNegative numberChemical equationSoftwareQuicksortType theoryDifferent (Kate Ryan album)FrequencyInternet forumConstructor (object-oriented programming)PredictionComputer animation
20:46
Metric systemEndliche ModelltheorieMetric systemIntegrated development environmentRight angleBit rateSet (mathematics)Real numberFLOPSSystem programmingAutomatic differentiationSource code
21:32
Information privacyExploit (computer security)Different (Kate Ryan album)Level (video gaming)Self-organizationQuicksortEndliche ModelltheorieLatent heatRight angleInformation privacy
22:40
SicGEDCOMGoodness of fitSkewnessPurchasingMusical ensembleProjective planeAlgorithmMeta elementData conversionRight angleComputer virusComputer animationLecture/ConferenceMeeting/Interview
23:56
Data conversionBookmark (World Wide Web)MereologyRight angleComputer animationMeeting/Interview
24:31
System of linear equationsCross-site scriptingPoint (geometry)Multiplication signMeeting/InterviewComputer animation
25:16
Ordinary differential equationTime seriesCASE <Informatik>Computing platformQuicksortGraphics tabletTwitterRight angleSet (mathematics)Computer animationMeeting/InterviewLecture/Conference
27:08
CASE <Informatik>Internet forumRoundness (object)Presentation of a groupComputer animation
Transcript: English(auto-generated)
00:04
So, my name is Bumiaki Remi and I am a machine learning engineer. But then, just before I start, I mean, you probably all see this meme somewhere, maybe, and then you're like,
00:22
how does this work, how is it that? Before, maybe you can have a conversation with your friend and it seems like Google already saw it. And then you search for something or you just open your phone and that's the next ad to see. So, are they really listening to your microphone or are they reading your WhatsApp messages
00:43
or are they using your camera or are they reading your mind? These are probably questions that we've had at one point or the other, but then we all, well, I think we're all technical guys, almost all of us are technical guys,
01:00
and they're like, it's not possible to read my mind. So, what really is going on? And that's what gets debunked in this session today. Okay, so I'm a machine learning engineer and surprisingly, I've recently found interest
01:20
in a couple of areas like policy, economics. Why? I don't know. I think I'm just a pretty curious person and I would like to have like a broader view of the world. I mean, I'm not doing a PhD, so I probably don't have scenarios in a particular area.
01:45
And I also like to participate in hackathons, right? I like to hack a lot. And I'm also a community advocate. I super love what we do at EuroPython. I love the Python community, trying to contribute as much as I like.
02:03
And something that really caught my interest these past few weeks has been quantum machine learning. Like, yeah, I really don't know why, but I met this friend and instead of talking, I think it's probably an interesting area.
02:20
So, outside tech, I'm a poet, I like to write, and sometimes probably find me taking pictures, weird pictures, yeah. So, back against this photo, how do these big organizations,
02:40
how do they actually leverage our data? And how do they take, I don't know, do they actually track our every move? What data are they actually tracking? And how do that data translate to relevant ads?
03:01
And how are the algorithms so good that it can literally predict what you're going to buy next or what you're going to need next, or next phone that just suits you? So, how do they do that? Now, before I start that, I just want to give an overview of what the talk is going to be about.
03:23
Going to talk about data and ML ecosystem, and by data, I'm narrowing it specifically to ads prediction data. Also going to talk about some key concepts in ads prediction. And we're going to look at a design flow
03:41
or some design flow of an ads prediction system, how the various layers of algorithms and filtering that the data has to go to before you have a very fine-tuned ad that you're most likely going to click.
04:01
And how the advertiser and the advertising platform benefits from it too, because there is a side where you get to see ads and then there is the other side where an advertiser also gets the most benefits. And the advertising platform also helps make the most profit.
04:23
So, you have like 30 key players in this conversation. And of course, we need to also talk about some ethical considerations because there have always been questions and there would always be questions about, I might track every moment, right?
04:42
What data are they seeing? What data can I restrict them from seeing? And what are the policies around this? And also, I'll be open to take questions first. Okay, so the data ecosystem is,
05:03
you can come from its dimensional prospects even say between you and the advertiser. And by advertiser, I mean someone who wants to advertise a product, right? So, the advertiser creates an advertisement and uploads this to the advertising platform.
05:21
It could be Google, it could be Facebook. Probably some of us have tried to create advertisements ourselves. And then this goes to like, I can say repository where you have so many ads. So, you could have like millions of ads, right? And by ads, I mean like active ads, yeah?
05:43
Now, how does this sort of narrow down to you and get an ad that's specific to you in particular? So, you, a user, sends a query. Maybe you go to Google and then you search.
06:03
Maybe you search for a Lenovo tablet, for example, right? Or you search for maybe baby kids or just something. So now, how does that ad know that this particular brand is what you should get?
06:21
Okay, so now you send your query and your query goes to look for a set of related ads. Now, related ads, this could be thousands, like 100,000. It could be so many related ads. And then, predict the ones that you're most likely to click.
06:41
This could narrow it down to maybe 50 or 30. And then there is another system where the ads are actually auctioned, right? Auction is probably from here towards you, where you're like, okay, who's the highest bidder?
07:00
So, among these top ads, so which of these advertisers sort of pay the highest for it? So, of course, the more you bid, the higher your chances. So, then you probably come down with a list of maybe top one, top two,
07:24
depending on the platform that this ad is being displayed. So, you get your results and ads. So, this is just like a quick overview of what the data ecosystem is like. Now, the ML ecosystem is probably what some of us are familiar with.
07:44
You have data pipelines, you have ML models, you have where they are trained, stored, served and monitored. Now, if you're quite familiar with the ML system, you know that these things are continuous. So, for example, in this scenario, you have like so many models,
08:05
you can have like three models, four models, five, ten models, performing one single step in the whole ad prediction process. So, how do you efficiently manage training? Maybe you're training in real time or you're training near real time
08:22
or you're also managing like loads of data. So, for example, a platform like Google, has like so many users. So, how do you manage that influx of data while delivering on efficiency, cost and latency?
08:41
Okay, so, while you as a user, you're like, ugh, ads again, on the back end, there is like a whole lot going on, on how the data is being managed and narrowed down to you. So, if you're saying, okay, maybe this company X is actually tapping into my microphone
09:01
and listening to the conversations I have. So, I'm not saying that's true, right? I'm just giving a scenario. So, how do you think that company is going to manage, not just for you, but for all of the users possible, right? And efficiently track and narrow down that, okay, this is this.
09:21
So, that's not possible, right? And you're gonna ask questions like, for how long are they listening to you? So, some things are just like, not possible, because it's against data privacy, unless you're actually speaking to, maybe speaking to the product, you shouldn't, no company should be tapping into your microphone,
09:43
I mean, except you're being asked or something. So, yeah, in the ML ecosystem, there is a whole lot going on and there are questions around sustainability, efficiency, costs, right?
10:00
Because we have so many models doing so many things, right? And they're all in the pipeline. So, how do you get your data from the source down to that particular ad which is the most performing in the minutest amount of time, right? So, these are the questions that we answer.
10:23
Now, some key concepts in advertising, you have lead optimization, some you're actually familiar with, like ads ranking, cost per click, right? And some you're not really familiar with, like what's conversion rate?
10:41
So, if you're probably have like your own personal blog, you'd see things like conversion rate, so like who clicks it, how do they click it, where do they come from, from WhatsApp, maybe from search or whatever things. And then cost per acquisition, that is cost for acquiring a particular customer
11:03
or maybe not acquiring but for doing a particular thing. So, for example, your goal is, okay, I want this person to sign up on my app. So, I'm putting an advertisement for people to sign up on my app. For some other people, I'm putting an advertisement for people to buy this product, right?
11:22
So, it could have like different goals. So, there is also bid optimization, where you're trying to determine the optimal bid price for an advertisement placement, right? Which is part of what we do at Kochava, help you sort of figure out what projects do you allocate for particular advertisers.
11:47
So, now you're also asking what data do they track? They track the graphics, your age, your gender, the language you speak, those kind of things.
12:03
They also track your browsing history, passport changes. I mean, some of these things you probably see them in your Google profile settings, right? Now, that's you. They also track other things about the ads itself. So, it's previous impressions, clicks, conversions.
12:23
So, they want to know the history of these ads, how has it performed? So, if I'm recommending an ad to you and it performs really bad, right? So, that says something about that advertisement itself, yeah? So, maybe it's not the right, it's either it's not being presented well
12:43
or, you know, there's some other questions around that. So, because the advertising platform wants to make sure that I'm getting the most out of whatever ads I'm getting, I'm pushing out, yeah. So, they also track other things like the contextual data, the ad placement data,
13:06
that is what network, who's the publisher and all these kind of things. Okay, so, yeah, let's look at the ads prediction system itself.
13:20
So, this is just like a more condensed view of what I showed before. So, this is the graph I showed before where you have the user and you have the advertiser, right? And now, there are different data types when it comes to ads.
13:48
You probably had me mention that there are specific data related to the ad itself, to the advertiser who is advertising, to the user, right? And then you have some cross features between the ads, the advertiser and the user
14:04
and also context. So, context is also really important. These are like the important key feature types and you should also be able to identify them. So, for some things like campaign ID, publisher ID, network ID and all those other kind of things.
14:23
So, these are like some of the feature types. So, now, how does this actually work? So, you're the user, right? Maybe searching through social media, Google, whatever search platform. I mean, and you don't always have to search.
14:41
So, this is just like a limited sample. So, your query, either your query is clicking on something or typing something or sending a voice command, it's a query. Then this gets combined with the context, the user demographics, maybe what any other data that the advertising platform chooses to add.
15:06
And this goes through the first phase which you can call like selecting related ads. So, first of all, you want to check, okay, what ads are related to this particular query that I've constructed.
15:22
So, your first batch can be maybe 100,000, 10,000. So, your first batch can be like, can be so much. And then, okay, so imagine if I have like 5,000 ads.
15:41
My next question is, okay, how do I select which of these 5,000 is the best? Yeah. So, in the first stage, maybe you want to use a less complex model, like a simple model like just you can get shown. So, first of all, filter that out because it's simple, it's fast,
16:04
doesn't need like too much complexity. So, the place where you need more complex models are where you're trying to select the relevant ads. And also predict the high-performing ads.
16:20
So, you have like 5,000 related ads. Now, which of these 5,000 is actually relevant at this time with this context and this critical item graphics and other questions that you have around it. So, now, yeah, so now you have like, maybe this narrows it down to about 100.
16:45
Yeah. So, now, the next stage now is which of these 100s would actually be high-performing. So, you have a more complex model to actually filter this out to maybe 10, right?
17:02
10 or 5, really, it depends on you. And now the most interesting part is where you actually get option. Now, this is where the advertising platform wants to get the most, right? The advertiser wants to get the most and they want you to click the ads.
17:22
So, which ads is going to sort of fulfill all that criteria? And that's where the ads actually come in, right? And they rank the ads by the ISV data. This ISV data is high-performing, right? The ads are high-performing.
17:41
And if it's not a new ad, maybe it's like it already has like some performance data. In the past, these ads maybe performed really well, right? And then it's offering maybe almost the highest amount of money and then they push this out to you.
18:00
So, this could be maybe you sit on your laptop or maybe as a pop-up or whatsoever form. Now, the platform in which you see these ads also determines the number of ads you see. So, sometimes maybe you see one, sometimes you see two. Maybe like YouTube, you'd see one maybe in the midst of like two videos, right?
18:22
It just depends on... Yeah, and also the platform also determines what kind of ads you see. So, the format of the ads, right? Remember I talked about ad-specific feature. Is it a video, is it a text, is it an image, right? That also determines the platform
18:40
and that also determines whether you get to see it or not, okay? So, some information that I didn't include in the graph is online training. So, because of these kind of systems, they get like easily outdated.
19:01
So, you need to keep training and training and training and training and training. And that frequency depends on you, right? Because you're always getting data, right? From the user, from the ads' performance itself. So, maybe these ads get pushed and what's the performance, right?
19:22
So, you're always getting data. How often do you want to retrain your model? And there's also something you probably want to be aware of, right? Because if you have like so many negative samples, right?
19:41
And you have like fewer positive samples. So, how do you sort of balance the model so that the model doesn't go biased towards like negative samples, right? That's where you need to do some recalibration. In the asset selection, you talk about constructing query
20:01
which is also really important. And what are the other factors? The other things that probably I didn't mention here, right? That's also where you're trying to like construct a query. And ads prediction, predicting user engagement. So, there are actually a lot of questions that you need to ask around these systems.
20:21
So, it's not like you're just going to take what I say and you're just going to get... No. There are a lot of questions that you need to ask not just with the developers, with the stakeholders. I mean all forum stickers from the executives to the advertising platform. So, all these come in the whole system.
20:46
Okay. So, just to quickly talk about metrics. So, for people who haven't worked in production environments before, you're probably familiar with accuracy, precision. Yeah.
21:01
But then in the real world, you're looking at a whole lot set of metrics entirely. Looking at click-through rates, looking at what's written on ad spends, brown streets, lift metrics, engagement metrics. Because all this really determines if you're pushing out the right ads or there is a problem somewhere.
21:21
And that's why this needs to continually be monitored so that you can figure out if there's a flop with the model or if there's an issue somewhere you can easily meditate this. Okay. So, ethical considerations, privacy, right? I know that organizations are now fighting for anonymity
21:44
and things like that. But then even anonymity has its own levels, right? So, different levels to anonymity. So, I think that every country has their own data guidelines. So, it's best to sort of stick to the regulatory compliance
22:02
specific to each country. And as much as possible, it's good to have a very transparent and accountable model, right? So, make sure that your model is transparent. You know what's going in, you know what's going out, you know why. And you also want to avoid manipulation and exploitation
22:21
because it is very possible where you're manipulating the user, right? While trying to get the most profits, you're actually manipulating them to go towards, for example, elections, yeah? So, you're manipulating the people to go towards the candidate.
22:41
So, thank you so much. I hope you've learned a lot of good things. And we have questions. So, thanks, Munmi. And if you have any questions, please come to the mic.
23:02
Hello. Thank you for the talk. I was wondering if you have any suggestion to get the right return on ad spend from Meta because I'm working on a project and it gets sometimes very muddy and we're not being able to get that metric right
23:23
because it depends on the amount spent and the purchase conversion value and sometimes it goes really high, etc. That's one question. And the other one is for unbalanced data inside your dataset, if, for example, you got zero clicks for a campaign, let's say,
23:46
do you balance the data for your algorithms or what's your suggestion on that? Thank you. Okay. For the first question, I don't think I might be able to answer it here in public
24:01
because it's also part of what I'm doing at Kuchawa, so like confidentiality. Maybe you can reach out to me and we can have like your favorite conversation on that. Okay. So, the second question, well, I think it depends. For example, if it's just clicks, right,
24:23
okay, yeah, I wanted to ask, so is it that this is like an entirely new ad? Yeah? Actually not. They've been running the different campaigns for about four years, I think. Okay, so this particular ad running for four years
24:41
but still has zero clicks? Yeah, like at some point, let's say for some particular days because we get the data points up to a day to see which campaign, for example, is the winning campaign, which campaign has, for example, zero clicks at some time,
25:02
some particular day, for example, and then we try to get or to predict which one is going to be like the best campaign. Okay, okay. So, if it was an entirely new ad, I would have said maybe you sort of add some values
25:23
to make it not zero because, of course, a new ad has no history but if that's not the case, you want to look at if clicks is the only metric with zero, you can also use some of the historical data. So, it's not like it's isolated.
25:43
So, if it's zero, that's telling you a lot about how that ad is actually doing and if you should be pushing that ad to that particular set of users or customers you're pushing it to. So, you probably want to ask yourself questions around if we're pushing
26:00
the right things to the right people, right? Because I wouldn't say the best thing to do is to pad it up, right? Because that alone, like, chose those sort of questions. So, why do you have zero clicks? So, that's the first question and if you probably have an answer to that,
26:22
if it's something that maybe the platform you're showing it to does not allow users to click or, you know, there are other questions. So, I initially would not suggest adding values to make it like the rest because that says a lot.
26:42
But again, you said that some days you have zero, some other days you have... I think that's really the way time series data is. You know, some days you have some more, some days you have some less, but again, you also want to ask questions as to why you're seeing that trend. I think that's going to inform what you do next.
27:02
But I would not default saying add data to make it look like the rest. Thank you. Are there any questions? Well, in that case, I would like to thank again Purnima for the wonderful presentation
27:21
and let's give a round of applause.