A smooth ride: Online car buying and selling at mobile.de
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 56 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/67153 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Musical ensembleBitXMLUML
00:25
Complex (psychology)Decision theoryComputing platformRight angleProcess (computing)Source codeBitElectronic mailing listLecture/ConferenceComputer animation
01:31
Process (computing)Line (geometry)Expert systemLecture/Conference
02:02
Computing platformFamilyMoment (mathematics)Process (computing)Expert systemElectronic mailing listPoint (geometry)Flow separationBechtle AktiengesellschaftComputer animation
03:57
World Wide Web ConsortiumTurbo-CodeProduct (business)PredictabilityDifferent (Kate Ryan album)Point (geometry)BitVirtual machineEndliche ModelltheorieDecision theoryElectronic mailing listComputing platformDigitizingLecture/ConferenceComputer animationDiagramProgram flowchart
05:06
NumberElectronic mailing listProduct (business)Virtual machineQuery languageFreewareProfil (magazine)FamilyEndliche ModelltheorieMereologyTurbo-CodeDifferent (Kate Ryan album)Lecture/ConferenceComputer animation
06:34
Product (business)Computing platformPredictabilityEndliche ModelltheorieBitGoodness of fitProfil (magazine)WordLecture/ConferenceMeeting/Interview
07:33
WebsiteBitRankingWordOrder (biology)Lecture/Conference
08:01
FrustrationResultantElectronic mailing listComputer animation
08:25
RankingInformationData modelLambda calculusMetric systemElectronic mailing listMoment (mathematics)QuicksortMultiplication signProfil (magazine)Level (video gaming)WebsiteVirtual machineInteractive televisionCASE <Informatik>MultilaterationResultantWeb pagePosition operator
09:23
Interactive televisionQuicksortUser profileCorrespondence (mathematics)1 (number)Wave packetLecture/Conference
09:55
InformationMetric systemData modelLambda calculusRankingFingerprintPlug-in (computing)Endliche ModelltheorieFeedbackLambda calculusHeuristicType theoryProcess (computing)Plug-in (computing)Physical systemGoogolIntegrated development environmentProduct (business)Cloud computingComputer animation
11:01
Plug-in (computing)Execution unitRankingType theoryUser profileEndliche ModelltheorieService (economics)Meeting/InterviewLecture/ConferenceComputer animation
11:25
Elasticity (physics)Query languageData conversionInformationData modelMetric systemLambda calculusProgrammschleifeDefault (computer science)User profileMusical ensembleQuicksortRankingWeb pageDefault (computer science)Profil (magazine)Endliche ModelltheorieLecture/Conference
12:06
RankingQuery languageElasticity (physics)Data conversionInformationLambda calculusMetric systemData modelProgrammschleifeDefault (computer science)CASE <Informatik>Software frameworkEndliche ModelltheorieObject (grammar)Lecture/ConferenceMeeting/Interview
12:31
FeedbackResultantObject (grammar)Virtual machineEndliche ModelltheorieLecture/Conference
12:51
RankingElasticity (physics)Query languageData conversionInformationData modelLambda calculusMetric systemProgrammschleifeDefault (computer science)EstimationMultiplication signPredictabilitySet (mathematics)ResultantProduct (business)Lecture/Conference
13:21
EstimationJava appletQuery languageInformationData modelAbsolute valueMetric systemArithmetic meanError messagePredictionConsistencySingle-precision floating-point formatAbsolute valueEndliche ModelltheorieScaling (geometry)Arithmetic meanThresholding (image processing)PredictabilityWeb pageTerm (mathematics)Multiple RegressionVirtual machineFehlerschrankeElectronic mailing listEstimatorGoodness of fitMultiplication signStandard deviationComputer animation
14:35
PredictionEmailConsistencySingle-precision floating-point formatQuery languageMultiplication signStack (abstract data type)Endliche ModelltheorieCASE <Informatik>Pairwise comparisonProduct (business)Software developerMereologyElectronic mailing listBranch (computer science)PredictabilitySoftware frameworkAxiom of choiceProcess (computing)GoogolJava appletPoint cloudInformation engineeringRepresentational state transferService (economics)Data storage deviceConsistencyLecture/ConferenceComputer animation
16:29
PredictionJava appletQuery languageInformationAbsolute valueMetric systemArithmetic meanError messageData modelConstraint (mathematics)AerodynamicsBitEndliche ModelltheorieDefault (computer science)PredictabilityShift operatorDynamical systemDistribution (mathematics)Goodness of fitPlastikkarteDirection (geometry)Instance (computer science)Lecture/ConferenceMeeting/Interview
17:31
PredictionEndliche ModelltheorieBit rateInstance (computer science)Theory of relativityBitLecture/Conference
18:06
Constraint (mathematics)FeedbackEndliche ModelltheoriePoint (geometry)Lecture/Conference
18:43
Product (business)BitComputing platformLecture/Conference
19:03
Computing platformQuery languageMetadataInformationData modelMetric systemService (economics)Software frameworkMereologyMetric systemProduct (business)CASE <Informatik>Different (Kate Ryan album)MetadataElectronic mailing listWave packetService (economics)Basis <Mathematik>Endliche ModelltheoriePredictabilityDecision theoryInformation engineeringPresentation of a groupComputer animation
20:42
Endliche ModelltheorieService (economics)MetadataWebsiteComputing platformQuery languageInformationMetric systemData modelPerformance appraisalPredictionDecision theoryComputing platformPoint (geometry)Multiplication signMathematical analysisChemical equationPosition operatorVolume (thermodynamics)Software testingService (economics)Insertion lossWave packetPredictabilityDatabaseJava appletElectronic mailing listProjective planeShared memoryView (database)FlagEndliche ModelltheorieLevel (video gaming)Spring (hydrology)Representational state transferSet (mathematics)Confidence intervalThresholding (image processing)Lecture/ConferenceComputer animation
23:23
Decision theoryMultiplication signElectronic mailing listService (economics)Black boxPredictabilityLecture/Conference
23:52
InformationMereologyFront and back endsGoodness of fitInformation engineeringDebuggerLecture/ConferenceComputer animation
24:23
Elasticity (physics)Multiplication signLecture/ConferenceMeeting/Interview
24:50
RankingMultiplication signRevision controlEndliche ModelltheorieSoftware testingLecture/Conference
25:17
Open setStreaming mediaBitPoint (geometry)LogicWeb pageAreaQuicksortEndliche ModelltheorieDigitizingVirtual machineComputer animationLecture/Conference
26:07
Musical ensembleLecture/ConferenceJSONXMLUML
Transcript: English(auto-generated)
00:07
Don't need to clap yet, but thank you. So, guys, we are from Mobile.de. My name is Ricardo. I'm with my colleague, Marlena. And we are here to show you a little bit what we do at Mobile. It's a sponsored talk. I will need to tell
00:24
this upfront so we could choose what to present. So we're going to try to present to you an overview what we do with the data without revealing too much of our competitive advantage, right, on the market. So if you live in Germany and you own a vehicle, most likely you have heard from us. Mobile.de is the leading
00:44
platform of buying and selling vehicles online. We have 12 million users, active buyers per month, 1.6 million listings online on any day. And our main source of revenues are our dealerships, which are paying customers
01:02
and currently we have a little bit more than 40,000 dealerships that are customers of Mobile.de. And Mobile.de supports the process of buying and selling vehicles online. And it's more complex than any other good that you might try to buy online because it's a high-value good, which comes with some
01:26
implications there. Because for many people, it's a stressful situation when you're trying to sell a vehicle or you want to buy a vehicle because a lot of money involved, a lot of uncertainty. For that reason, the journey to buy or sell a car looks more like this. It's not a straight line. And there are a lot
01:44
of questions that come up for a regular user during this process of selling or buying a car. First thing, how much is my car worth? If I want to sell my vehicle, I'm not an expert. I'm not sure how much the market is willing to
02:02
pay for it. A second question that might come to you is how fast I want to sell my car. Do I need the money right now or can I take longer to get a better deal for my vehicle? Another question, what's the right audience that I need to target to sell my vehicle? Do I publish only at Mobile? Mobile.de has a strong
02:23
partnership with Bechtle & Zeigen where we can publish the listings there. It's a question that might come to you when you're selling. Another one, it's like Mobile.de is a platform that offers extra booking features that might increase
02:43
the visibility of your listing. Do you need that? It's a question that at some point you might ask yourself if you're having difficulty to sell it. Then when you go to the buying process, the first question, what's my next car? How do I find it? Which car actually best fits my needs? I might not be an
03:03
expert of knowing how to make models, so I'm not sure which one fits my need if I have a family, if I need a sports car and so on. Finally, how exactly I find this online after I decided which car that best fits my needs? Of course, after you have everything set up, the main question is
03:24
which seller offers me the best offer that's the best deal for me? Finally, at the moment you found the seller that you want to open a negotiation, can you trust it? Can you trust the seller? Usually online marketplaces have a
03:40
severe problem with online fraud and there is a question that might come to you. Finally, at the moment that you want to finalize your deal, how do you do it? Do I need financing? Do I need some extra money? How do I close the deal? There are lots of questions that come up in this scenario. For
04:00
each one of those questions at Mobility, we have a data product on it. What does it mean, a data product? We have a machine learning model that supports the user in making those decisions at these points during the journey. I'm going to try to cover a little bit of each of them. For the first question, how much
04:24
is my car worth? We have a price prediction model, which means with very few features of the vehicle, we can already suggest what's the price of your vehicle. The second thing, do I want to sell fast or do I want to get more money for my vehicle? We have a recommendation for different products in our
04:44
platform that can lead you to sell fast, so we can connect you directly with a dealership that might be willing to pay a certain value for your vehicle or another product that's to publish your listing online and try to get more money out of it. To target the right audience, we have what we call the Digital
05:05
Market Hub, which is a product that, given your vehicle and the features, we can find out for which audience this is most attractive, so how we can improve the number of leads you get for your listing online. Also, for this product of
05:28
booking extra features, we have what we call Ad Turbo, which is a product that evaluates your listing and see if any of the features can have a positive impact on the number of leads you get for your vehicle. For the buying part, we have,
05:48
of course, search and ranking for finding the vehicles. We have vehicle recommenders, so given your profile, we can recommend you different types of vehicles if you need an SUV for a family or sports car, as I mentioned. How to find
06:04
the best car? We have also recommenders for queries. Our query is not free text query. It's rather structured, but sometimes the users, they have problems in defining which filters to use to find their vehicles. We have a product that suggests those queries. Finally, not finally yet, but for the price, how do
06:25
you find the best deal? We have another product, another machine learning model to predict the market prices. We not only predict the price for a given vehicle, but how it compares to the whole marketplace, the data that we have, so we can see if it's a good deal or it's too expensive, the vehicle, and so on.
06:45
Finally, we have models to detect fraud in our platform if a seller trying to deceive the buyers and also if the sellers lost their account due to account takeovers. The last one we have is the prediction of financing intent, so given
07:02
a buyer and his profile and his data, we can figure out more or less if the user, it's a user that would require financing to close the deal, and we suggest for this user to go into our partner's bank that would support the user getting financing
07:24
for the vehicle. It's a broad overview of all these data products, and we're going to take a look a bit deeper in some of them, and I give the word to Marlene. Thanks. Yeah, so the first one that we chose for this deeper look is our search
07:42
ranking because we thought that makes sense also given the audience, probably nothing I would say is new to you, but still I would like to present our approach to this. So generally, when you come to our site, you are interested in finding a car, but in order to find a car, you first have to search for it. Even if you know already a little bit what you're searching for and you have
08:03
some search criteria in your mind, there will probably be a lot of vehicles or cars that fit these search criteria more probably than you can even scroll through. So wouldn't it be nice if we could somehow put those items first in the list of your search results that are also most relevant to you, so most personally
08:23
relevant. And we try to achieve this by using like a learning to rank approach, so a machine learning approach. For this, we take historical data on searches. Basically, we try to create snapshots of the moment in time when users search in the past. So we know which listings were shown to you and also at what position
08:45
they were shown to you. We know other features about the listings that were displayed in the search result page. We know whether you interacted with these listings, as in whether you clicked on them or you maybe even parked an item, which in our case means you moved to some basket for later, or even whether
09:05
you contacted a dealer. So we have these levels of interaction basically. And we also have some sort of profile on the user at the moment of their search, which is however just like an aggregation of the user interactions with our site.
09:20
So we know like in the past two weeks a user maybe has interacted mostly with blue cars or something like this, so there's some sort of user profile that we use for this. And then we create like a training data which corresponds of features that come from the car itself and features that come from the user, features that kind of combine the car and the user and how well they fit. And then we try to rank items in a fashion
09:45
such that the most relevant ones will be placed on top. For this we need some sort of target or label, right, the relevancy signal. And here we take it from this user interaction. So we say if you clicked an item it's more relevant compared to when you didn't click it. If you parked it, it's even more relevant. So we come up with this heuristic of creating
10:05
labels that are based on this implicit feedback that the user gives us. Then we train a learning to rank model based on these features and target. We use a LambdaMart model, at least I think in the last model it was LambdaMart. And then we try to optimize for NDCG here.
10:24
How do we do this in practice? Maybe how is this implemented in our environment or production system? So we use Google Cloud Platform mostly. And so we try to do as much of the data querying in BigQuery. Then we schedule certain jobs in Python using Airflow.
10:43
This is mostly the feature processing. Then we also build the model in Python. And we then upload the model that we have along with the feature set into Elasticsearch. There's an LTR plugin which lets you directly use certain types of models
11:01
and certain types of features so that these would be online, passable and directly available basically as a user searches. So this is as a user sends a search request. It will go through our model and also we have a custom service or API that provides us aggregated user profile of the user currently searching.
11:22
And so we have basically a model that is running online and it's personalized because it can take a user profile into account. And I think maybe I would skip some of the things here, but I think what is most interesting about these models maybe is our specific learnings.
11:41
That we have from it. So since we try to achieve like a personalized search ranking, of course it's important that we have some sort of user profile. So there needs to be some sort put into how do you store or represent such user profiles and also how do you deal with users that have not visited your page before.
12:01
This is like the cold start problem. Do you create some default profile for them? Which is what we do. And then also in our case, so I presented only this really schematic framework of how this is designed, but usually we don't only want to optimize for the user satisfaction,
12:21
so whatever is relevant to the user, but we also have some business requirements like sometimes dealers pay maybe for certain features and they want to be ranked higher. And so we also tried to combine multiple objectives basically in our model so that we can account for both of these interests directly within our machine learning model.
12:42
And of course, we have to watch out for biases since we use this implicit feedback of the users. We know that the behavior of the user is already influenced by however we displayed the results to begin with. So we have tried some approaches to correct for these biases, but I'm not going to go into detail about this.
13:00
Do we have more time? So this is our search deep dive. This is how we try to personalize your search results at Mobile. A very different but also really cool product is the market price prediction. So now let's say you already have like a candidate set of certain vehicles that you find interesting, but you don't really know if the price is appropriate or not.
13:23
You want a fair price and we try to guide you in finding a fair price by providing this little UI, as you can see here. So for most vehicles, you will see a prediction or estimation whether the vehicle that you're currently looking at is actually a fair price or not.
13:44
Again, this is based on a machine learning model. We train on historical data, on listings and their final prices when they were sold or at least deleted from our page. Then we train an XGBoost regression model. We evaluate those models in terms of mean absolute percentage error.
14:05
And basically, in a scenario to get to the scale from very good to high price, you would compare the price that has been inserted by the dealer maybe to the price that your model predicts. And then based on some threshold of deviation,
14:22
you would come up with a label of it being a good price or not so good price. I think maybe because of time reasons, I should I skip this? I think we're good on time. Yeah, OK, because last time we were so over time. So the tech stack that we use for that is very similar, I think,
14:43
to the LTR model that I presented earlier. So we try to do as much of the querying in BigQuery from Google Cloud. And then we schedule a couple of jobs in Airflow. In this case, actually, a lot of the feature processing and modeling is happening at R. This is just by choice of our data scientist here.
15:03
And then we upload an H2O model into the Google Cloud Storage, where it can be retrieved from a custom built REST API in Scala. It can pass this model. So this is why we use H2O as a modeling framework
15:20
because it provides Java and Scala passable models, basically. So our data scientists build the model, and then a data engineer would try to also just recapitulate the features that were built, but then the model would be directly available for production. And then within the service, we generate these online predictions
15:41
that are available then as the user searches again. Each time that a listing is changed, maybe a feature is changed, it also triggers that the prediction is updated for the particular car. We have another branch here, maybe in this pipeline layout,
16:01
to just demonstrate, of course, we always have to deal with consistency between what we do in the development part and the production part, so to make sure that everything that's happening in this REST API, which is more Java-based, and the development part, which in this case is R-based, we also automatically create certain requests to this API
16:25
and get the predictions and compare them, basically, to the developmental predictions. This is a little bit of a manual approach. This model is updated every three months or so, so we typically monitor KPIs. In the meantime, to see if something really deviates, for instance, if the distribution of good and bad prices, et cetera,
16:44
if this really shifts in one or the other direction, we would also, of course, update the model, but generally three months is just practically feasible and enough to account for most of the dynamics in the market currently. What are the learnings from our market price prediction model?
17:03
So this, of course, works best for cars for which we have a lot of reference cars. So if you have a very, very particular car that has been maybe even, I don't know, pimped up a specific way for your own requirements, then there would probably not be a reference car to make such a price prediction.
17:21
So typically, we restrict that model to kind of default cars, and you have to select specifically which cars can go into these models. Also, another learning is maybe that sometimes model, of course, can behave in a way that it's not intuitive to, for instance, a seller or dealer. So say that a seller enters all the features of a car, and then he gets a price rating,
17:46
and then maybe the dealer just adds that the car has a radio or something, and all of a sudden the price drops or becomes less fair or too high. So basically having added a radio as another feature makes the car cheaper,
18:03
which is a little bit counterintuitive. So maybe the model has picked up on some relations here, maybe it has been overfitting to something, I don't know, at this point, but definitely it's something that doesn't seem intuitively right to the user, and we try to account for this, for some features at least, by adding monotonic constraints to the model so that it's forced to learn
18:22
a specific, say, monotonically increasing relationship to a specific feature. So it's not allowed to go down in price or up in price as it wants. So this was an interesting learning, I think, based also on a lot of feedback from people interacting with this. So, yeah, then we have another deep dive into another data product,
18:46
which I don't know much about. Yeah, I can say a little bit about our fraud fighting initiatives. So the problem for us that every online platform has also is we are targets from scammers who try to get money out of the sellers
19:05
or trying to hijack their accounts. And our challenge is how to predict or detect when a seller is fraudulent. And for this, the data we have, we have historical data of past listings that have been manually evaluated by our customer service agents,
19:23
and they know if it's fraud or not. We use this to build our data set for training our models. And, of course, we have the data from the user that was fraudulent, how is the behavior, and a lot of metadata. So in our case, we are tracking a lot different from what the keynote speaker presented.
19:44
We are tracking for the good of it, so try to protect our users. The model and metrics you're using, we use precision and recall as metric. And it's a model stacking, actually, that we are using. The technology is very similar to what Marlene presented.
20:00
You're using GCP BigQuery. A lot of our data is stored in Mongo due to the decision from the engineers to have this in production. And we also use H2O framework to provide this flexibility for data scientists to work in the creative part and data engineers to take care that the service is responsive in production.
20:22
Because all these predictions, they are online predictions. Our pipeline here, we assume that the behavior of the fraudsters changed on a regular basis. So we retrain our model once a week, including new data to capture the new behavior of the fraudsters.
20:43
The pipeline is pretty much the same here by decision of the data science. They implemented everything in Python using Airflow on the GCP platform. The model is then exported to a Mongo database that can easily be loaded from the Java service,
21:01
which is the REST API that provides the predictions whenever there is an edit or insertion of a listing in our platform. When there is a prediction, then if we are certain about the prediction with a high level of confidence, we can automatically block or delete a listing and the user itself.
21:20
If we don't have this high confidence, but it's still suspicious, then we send to customer service and they would manually evaluate and give us feedback, which is the data that we use for training. There are other ways that we collect this data. So the data is not totally biased by the prediction of our models. Users, they can go and they can flag listings
21:44
that they think it's fraudulent. And these would also go to customer service. And then we get this data that's not biased by our predictions. One of the KPIs that we are using lately is the risk exposure.
22:00
So the share of users that come to our platform and they're exposed to fraud. So they had any view in a listing that was later assigned or detected as fraud. I'm very proud to say that this metric is now below 1%. So it's very low comparing to other platforms, other marketplaces
22:21
of the percentage of users that are exposed to fraud. Some of the learnings that we had in this project is that we need to arrange our data in a time-wise fashion because we cannot use the behavior of the dealer, sorry, the fraudster in any particular given time.
22:42
So our data sets, our training data, we have to split the training and test by time. Another thing that we have to be careful when considering is the impact of our false positives detections because this might harm our business, also bring a lot of volume to customer service agents.
23:03
So that's something that our thresholds, we are constantly evaluating and doing analysis to see if we keep the right balance at a given point in time because there is also a lot of seasonality. People usually sell their vehicles, they choose a spring to do that. It's when we have a high volume in our platform.
23:24
And finally, one thing that we had to invest a lot of time is to explain the predictions that we do for our customer service agents. It's not just a black box that the agents would see, these listings fraudulent, but they don't know why. And lately we are using some techniques
23:41
that can support our customer service agents to make those decisions. How am I in time? Pretty much so. I will skip the second part. It was about accounts takeover. I will invite you to find us in our booth.
24:01
We are sponsoring, we have a booth at the end of the room here. Our data scientists, I think we are four or five data scientists here from Obile attending the conference. We are happy to answer your questions if you come to our booth. Of course, we are also recruiting, always looking for good talent in data engineering, front end data science.
24:20
And I guess that's it. Thanks for your attention. Thank you very much. Thank you. We have time for questions or any short question right now or as announced later in the booth.
24:43
It's all right. Anyone? There's one question over there. Thanks for your insights. I would have one question regarding your learning to rank model in search ranking.
25:01
Could you share some insights about the uplifts you generated when going live with learning to rank? Oh, this is a long time ago, to be honest. So we ran some A-B tests on our first version, and I think we saw some uplift. I think our KPI usually is something like users with contacts.
25:23
At least this is like our North Star kind of. And we did see some improvement in the one digit area, like I think six, seven percent. But I should also say that this is compared to a non machine learning based ranking that we had previously. So previously we really had some sort of business driven logic of sorting
25:45
where the first page was basically just premium dealers and so forth. So what I'm saying is that I think the baseline was a little bit easy. Usually now when we try to iterate over the model, we typically just see like really small improvements and like one or two percentage points.
26:02
Thank you. OK, thanks. Thank you very much.