Scaling an online search engine to thousands of physical stores
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 56 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/67177 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
Berlin Buzzwords 202227 / 56
22
26
38
46
56
00:00
UnternehmensarchitekturEDV-BeratungDienst <Informatik>SoftwareentwicklerFormation <Mathematik>InformationsspeicherungBenutzerbeteiligungPhysikalismusEDV-BeratungNeuroinformatikBus <Informatik>ExpertensystemSuchmaschineTexteditorOffene MengeXMLUMLVorlesung/KonferenzComputeranimation
01:04
Singuläres IntegralUnternehmensarchitekturEDV-BeratungDienst <Informatik>SoftwareentwicklerArchitektur <Informatik>IndexberechnungProdukt <Mathematik>TexteditorUnternehmensarchitekturMultiplikationsoperatorUmwandlungsenthalpieVorlesung/KonferenzComputeranimation
02:00
SignalverarbeitungCASE <Informatik>UmwandlungsenthalpieInformationsspeicherungEreignishorizontMultiplikationsoperatorBenutzerbeteiligungComputeranimation
02:38
Produkt <Mathematik>PlastikkarteElektronische PublikationLokal kompakter RaumBeschreibungskomplexitätArchitektur <Informatik>IndexberechnungInformationsspeicherungSignalverarbeitungAutomatische IndexierungProdukt <Mathematik>Dienst <Informatik>Komplex <Algebra>DatenmodellSystemplattformSuchmaschineTopologieVerdeckungsrechnungDifferenteStreaming <Kommunikationstechnik>MAPNeuroinformatikBildschirmmaskeSoftwareDatenbankBitUmwandlungsenthalpieIndexberechnungGrenzschichtablösungPlastikkarteRelativitätstheorieElektronische PublikationCMM <Software Engineering>Vorlesung/KonferenzBesprechung/InterviewComputeranimation
06:56
Produkt <Mathematik>InformationFehlermeldungKomplex <Algebra>Automatische IndexierungIndexberechnungElement <Gruppentheorie>Stochastische AbhängigkeitArchitektur <Informatik>SystemplattformCMM <Software Engineering>SystemprogrammierungDatenverwaltungGruppoidComputersicherheitCybersexDigitalsignalDienst <Informatik>Automatische IndexierungProdukt <Mathematik>Rechter WinkelInformationsspeicherungCASE <Informatik>SuchmaschineIndexberechnungUmwandlungsenthalpieStochastische AbhängigkeitEinsDatenmodellPeer-to-Peer-NetzReelle ZahlPhysikalismusComputerarchitekturEDV-BeratungCoxeter-GruppeKartesische KoordinatenInformationSignalverarbeitungBitDienst <Informatik>MultiplikationsoperatorDatenstromSystemplattformTabelleVerdeckungsrechnungTypentheorieKomplex <Algebra>Kompakter RaumGüte der AnpassungAggregatzustandPhysikalisches SystemElement <Gruppentheorie>DifferenteMereologiePlastikkarteSchnittmengeKonfigurationsraumFehlermeldungTUNIS <Programm>DatenreplikationQuellcodeUnitäre GruppeExogene VariableVorlesung/KonferenzComputeranimation
15:59
Produkt <Mathematik>MultiplikationIndexberechnungArchitektur <Informatik>Zentrische StreckungStapeldateiComputersicherheitAutomatische IndexierungSuchmaschineKomplex <Algebra>SystemplattformMereologieQuellcodeBitTermIntegriertes InformationssystemDifferenteDatenstrukturPhysikalisches SystemIndexberechnungInformationVorlesung/KonferenzComputeranimation
17:22
Zentrische StreckungComputersicherheitStapeldateiProdukt <Mathematik>Online-KatalogKonfigurationsraumMultiplikationsoperatorSpielkonsoleComputerarchitekturComputersicherheitCoxeter-GruppeAutomatische IndexierungMereologieGlobale OptimierungInformationsspeicherungExogene VariableNichtlinearer OperatorStapeldateiGruppenoperationBitIndexberechnungProzess <Informatik>Data DictionaryProdukt <Mathematik>ResultantePunktDebuggingMomentenproblemWeb SiteKartesische KoordinatenElektronische PublikationSchnittmengeFacebookElektronisches ForumDifferenteSystemplattformMapping <Computergraphik>Dienst <Informatik>UmwandlungsenthalpieSoftwareentwicklerTUNIS <Programm>Vorlesung/KonferenzComputeranimation
21:53
Spitze <Mathematik>MehrrechnersystemRouterUrbild <Mathematik>KlumpenstichprobeArchitektur <Informatik>IndexberechnungHistogrammProdukt <Mathematik>TermExogene VariableStapeldateiOnline-KatalogChecklisteDienst <Informatik>SkalierbarkeitAutomatische IndexierungHackerComputersicherheitSchnittmengeMereologieAggregatzustandAutomatische IndexierungIndexberechnungInformationSkalierbarkeitAlgorithmusRouterBeweistheorieInformationsspeicherungResponse-ZeitLastSichtenkonzeptCluster <Rechnernetz>HistogrammChecklisteSuchmaschineBitGrenzschichtablösungEinsVirtuelles privates NetzwerkUmwandlungsenthalpieDatenstrukturSchnittmengeComputersicherheitFilter <Stochastik>TabelleRoutingTermDienst <Informatik>BenutzerbeteiligungProdukt <Mathematik>Front-End <Software>Web-SeiteResultanteVorlesung/KonferenzComputeranimation
27:11
Produkt <Mathematik>FehlermeldungComputersicherheitPhysikalisches SystemAutomatische IndexierungKonsistenz <Informatik>Vorlesung/Konferenz
27:41
RückkopplungWeb SiteOffene MengeSoftwaretestComputeranimationBesprechung/InterviewVorlesung/Konferenz
28:15
SpielkonsoleFitnessfunktionMereologieKonfigurationsraumSoftwaretestAbfrageResultanteIntegralAlgorithmusCodeBesprechung/InterviewVorlesung/Konferenz
28:57
ProgrammierumgebungKonfigurationsraumMereologieSoftwaretestMomentenproblemSchnittmengeIntegralElastische DeformationProzessautomationVorlesung/KonferenzComputeranimation
29:41
Response-ZeitZweiArithmetisches MittelMultiplikationsoperatorDebuggingMereologieKette <Mathematik>ResultanteDatensichtgerätFront-End <Software>Vorlesung/KonferenzBesprechung/Interview
31:09
Automatische IndexierungMultiplikationsoperatorInformationsspeicherungProdukt <Mathematik>BitDatenstrukturMereologieUmwandlungsenthalpieResultanteIndexberechnungInformationIdentifizierbarkeitPlastikkarteDeskriptive StatistikKategorie <Mathematik>TermVorlesung/KonferenzBesprechung/Interview
32:29
CodeStützpunkt <Mathematik>InformationsspeicherungGrenzschichtablösungAutomatische IndexierungDatenstrukturDifferenteFormation <Mathematik>UmwandlungsenthalpieKonfigurationsraumStreaming <Kommunikationstechnik>VersionsverwaltungPlug inMereologieProdukt <Mathematik>Vorlesung/KonferenzBesprechung/Interview
33:57
InformationsspeicherungRechter WinkelIndexberechnungAutomatische IndexierungDifferenz <Mathematik>GrenzschichtablösungSchnittmengeKonfigurationsraumÄhnlichkeitsgeometrieBitSchlüsselverwaltungElastische DeformationCodeMapping <Computergraphik>MereologieDatenstrukturDifferenteVorlesung/Konferenz
35:35
EreignishorizontSichtenkonzeptPunktZentrische StreckungBitMereologieStrömungsrichtungCluster <Rechnernetz>LastRouterAutomatische IndexierungAbfrageVorlesung/Konferenz
37:30
Formation <Mathematik>Vorlesung/KonferenzJSONXMLUML
Transkript: Englisch(automatisch erzeugt)
00:07
So yes, we'll be speaking about scaling when it's about physical stores, which is not exactly the same as full web search and full web stores.
00:23
I will start by introducing myself and the whole team at Adeline, because we are almost all here today at Berlin Bus World. So my name is Adeline. I am the CTO of this wonderful team. Well, we are experts in search engine technologies.
00:42
We are consultants with mostly Elasticsearch Solar, but whatever has search in it is interesting for us. So yeah, we do search. And we also are editors. Sorry, I just closed. This is not my computer.
01:01
You have a lot of things open, Omar. So yes, we are also editors of a solution named A2, which is a product that we sell and that we license, specific for e-commerce and enterprise search.
01:21
So we will speak about data when it's about the physical world. Then we'll talk more about deeply how it is indexed. Then we will see the story of an e-commerce competitor that grows, speak about architecture,
01:42
and run an infrastructure at the end. If you have any question, maybe it's better to keep them for the end of the talk. And I'm a bit late at the beginning, so I will try to speed up. Don't hesitate also to just interrupt me or tell me that I'm lacking time.
02:02
So let's go deep in the subject. We are in the physical world. So we are in stores with people in it, with their specificities. We are talking about any physical store that is going online that must do a digital transformation
02:22
or wants to, but with the recent events, everyone needed to get on the web sometime. So we are really, it's a specific case because there you have a high diversity of what
02:40
is in the shop, the prices, the stocks. Whatever is in the one online store will not be on the other one. So this is specific. Even if it's all under one brand, the grocery store and the physical retail is very specific.
03:02
Just to remember what are the expected features of e-commerce search. You probably know a lot about this, but maybe not. We expect search, of course. We need to find what we are looking for, find a product. We are also searching for autocomplete.
03:20
We need autocomplete. We need to access very quickly to what we are looking for. We also need to display a product card. We need to navigate through a tree, through the different departments. When there is no product, when we search for it,
03:41
we need similar products. We need suggestions. If we write something badly, we need a did you mean thing. So we need a lot of features around search. I will now explain to you what is happening
04:01
when a shop needs to go online. A shop handles a lot of data, which can be very, very data in any form. So you have files, maybe in folders in someone's computer. You have databases, you have the network, you have streams, you have different levels of maturity
04:23
in any store, and also you can find in such companies services. And the services can also all have their story, their data, their differences. And when it's about search engine,
04:41
you have to build a search engine above that. You have to deal with a lot of complexity coming with this data. But that's okay, because a search engine can really help a company that needs to go online in its digital transformation, which is good.
05:01
You need that, because when you have a lot of data that is very diverse and complex, you need to go very quick, and the search engine helps for it, because it eats almost anything, and provides this as a unitary,
05:22
and very common API or platform, which is very easy to use. It masks all the complexity, and for me, search can put all the digital transformation of a physical company one step ahead, and put it further, and maybe advance a lot.
05:47
So now a bit of technical things, because we are in the, so in the physical world, I say this, and we need to handle the how you index the different data we have
06:01
around the different products, the products data in every form. So we need to have indices, and we had to think about how you handle it. It's quite not straightforward. The first idea, of course, would be to have one index, and you put all your product referential in it,
06:22
and then as you know that you have specificities for offers, offers is like, we put the product in promotion, you put a price tag, whatever. So you have different offers, several per product, and you have different stores, and one store makes offers for products.
06:41
So you have a bit of a little relational data, you have a model in it, so you have to denormalize it, put it in an index, and then you're done. So it's easy, you have one index, you have a lot of products, a lot of offers, you have a very big index, and maybe it's a good solution, we'll see.
07:01
Another idea on the right would be to have one index with a referential of products. So we keep only the ID, the tag of the product, and then all the offers would be into a separate index. So one index per store with the offers
07:23
and the reference to the referential. Some of people maybe that work with search engines technically would see that there is already something that we do not really like. So to summarize, something like this, it's good, it's compact, it's unitary.
07:43
You have, like you denormalize, normalize data, but you can find your data model in it, you can stick to the data you expect to have. The cluster states would keep low, you only have one index, you have stuff very unitary.
08:04
But we found that you lose a lot of search performance because you have to search through large model. So this is the first problem we had. You have too much information retrieved, you really get, when you get to product,
08:22
you have a very big product card with all the offers and all that, you have to find in it what you need to display. Any small error in the product would break the whole thing. So if there is one product, one store,
08:40
putting a bad information, well, it will be replicated for everyone. There are frequent updates also because every time one store wants to change a price or do something, well, you will have to update the same document. And on the case two, you saw there was a join.
09:00
So joins on search, we don't really like that. So what we tried to do was to put one index per store. Let's do that. We'll duplicate, so we duplicate, we assume we duplicate common elements that are data that is common to everyone.
09:23
So we have a lot, a lot, a lot of data which is duplicated. We assume that. We put one index per physical store so that we can keep all the products, all the offers, the configuration. We can also go very deep in the settings,
09:41
the index settings are also one per physical store. So we really fine-tune everything for the physical store. And with that, we have, well, the cluster states increases because we have a lot of indices
10:03
and some updates, some stuff that happened, but the search performance is the best and it's really easy to get the product we need. Each part keeps its independence and when we are in the physical stores world, this is very important
10:21
because you won't say at all the store directors, the people that manage the thing, you will do the same that your colleague because they don't like that. So everyone needs to keep the independence and you have fewer updates per index. You have a lot of updates,
10:41
but they are spread with a different indices. So now, let's see another part. We are, let's say we have solved the problem of the physical stores. We assume that we are responding to all the specificities.
11:05
We are able to, well, respond to the needs of every physical store. Everyone is happy. And of course, well, what happened was really accelerated with the COVID-19 crisis in our case
11:21
because now everything accelerated. So the digital transformation really went faster. Every company wanted it and they matured very quickly. So what happened? Well, it is good,
11:40
but any company like that began to compete with the real pure players, the e-commerce pure players, the big ones. So there are other challenges that we'll see.
12:00
So this is the most beautiful schema of my presentation. I'm very proud of it. It's not mine. I should have credited it, but imagine, well, we are in the situation of the company when you bought the services of a wonderful company and a wonderful architecture consultant, IT services,
12:21
you know, IT design consultant, architect. And you know, you are finally maturing, converging. You have a lot of replication. You have architectures, you have services. You have, you know, you are in the pure players world. You have achieved your digital transformation.
12:41
You do like a plenary conferences and you know, you can say you are one of them. So the systems converge, this is good. This is good. You know, you are really thinking as a digital company. So this is good. You are in the full online world and this is normal.
13:05
What happened with the online, you know, shopping is that of course, when you are starting thinking like this, it opens a lot of doors in the mind. And anyone that succeeded into putting the physical stores
13:22
online can now pretend to be, you know, directly competing with Amazon or Rakuten or Alibaba or whatever in a country wide or even city wide. Why not? So this is real challenge and you can also imagine
13:43
that your search engine will now give them, give data and products from very little store. So you still have it. But at the same time, if there is no, not what you need here, you can also search into a marketplace. You can search into, I don't know,
14:02
a warehouse that provides speed delivery for non, you know, no food or stuff you need. So the idea would be that the search engine even masks more complexity. And now, you know, you have a search
14:20
which is really gets really powerful like the Amazon search, but including physical world. And also imagine that with all what we have into our hands now, we saw the keynote before, add a bit of personal data here,
14:43
personalization and it gets even more complex and more powerful. So the search engine is really, you know, very central. So yeah, we will have a search engine platform, one index per store. So a lot of, I said, I put a bit of ideas
15:04
of how many products you would find. So 20K products here, a marketplace that would be like provided like one data stream coming from, you know, another service. So you may have like 1 million, 2 millions products in it, unlimited.
15:23
A warehouse that contains products that can be delivered. So 500,000 or 1 million, why not, big indexes. And here you have another problem because this is not the same type of products.
15:42
You have here, for example, tomatoes, potatoes, you know, kids stuff, whatever. And here we will find iPads, iPods, you know. Here we can find a garden table, for example. So very different products coming from any data source
16:03
and you need to put a search engine platform above it that would be still performant. So what we are thinking about would be to have, you know, to work before indexing because we cannot add complexity to the search part
16:23
because we need to keep it, you know, speed quick. So we imagine like a common schema. We say that we prepare our data before indexing it so that we have a bit of a structure and the goal behind that is to make a search
16:42
through different indices, so multi-index searches. So once we have it, it has to work. And we saw that we have big indices, we have a lot of differences.
17:01
So we will now see a bit of, you know, how it can be built and how it can exist in a information system. So first let's see a bit of the need
17:21
in terms of infrastructure and what we need to address technically. First thing, we are in the mobile world. So we need that, you know, anyone shopping with a mobile phone will get tired very quickly. You can say it. When it's about one second, you know,
17:41
you search and it doesn't work, you will move away and check your Facebook updates or TikTok. So it needs to be very, very quick. And for the search part, we are challenged today to be under half a second for a response.
18:01
Then you have a lot of ingestion coming because for example, for a marketplace, you have batches of products coming all the time, all day long, every minute, I don't know, every five minutes and stuff. And you also have updates, you have removal, products removal, you have moderation because it's coming from basically anywhere.
18:23
And there are a lot of data processing, so you have to handle this ingestion. You need to provide search. You don't know what for. I mean, you have to build a platform, make it, I put fully integrated with the main front end,
18:44
you know, the website, the main website, the main application, but you have to be open so that anyone at any moment can develop something that would use the search in the whole services. So this is important to make an API to build it as a platform.
19:03
And to finish, well, you have to handle security, robustness, isolation of data. We'll talk about this quickly. So I'm sorry, it's a bit small. I don't know if you can read.
19:20
It's not very well written, but this is an example of, you know, the architecture, basic architecture using Elasticsearch. So this is what we do. So we have Elasticsearch with its configuration, the different mappings and settings for any index.
19:41
You will find one index per store, you will find the marketplace and other indices. You also have a configuration index and the configurations will contain also one config per index or per store or per specificity or per group.
20:00
To edit the configuration, we have a tool which you call the Business Console with the best Business Console developer just in front of me. I have to mention him because he saved the presentation. So big up the Business Console. So you can fine tune the configuration. You can really apply boosts, define the facets and all that.
20:24
It's very central and important to spend time on that because this is the entry point of all the business and all the people that know about the products. And then you have the two parts with a very, very important one, which is before indexing.
20:44
So there are data coming from anywhere. I didn't draw it, but you will find, you know, the files, the stuff, anything here. And before indexing, you have to do whatever it is that will speed up the search.
21:01
So all the optimization, all the, you know, any processing, dictionaries, you know, I don't know, anything that will help the index to be clean and to be searchable very quickly. So even if indexing takes a bit more time,
21:22
it can be complicated because we see that, for example, in the marketplace, I told you, it has to keep being fast. So there is a bit of a challenge here to be fast but performant, but the most important is that the search is very fast, always.
21:41
So in the search part, you also have some operations that you can do, for example, collapsing results or stuff like this, but keep it very unitary and very fast. So there are these two parts and the user will search. It comes here, easy.
22:04
As I said before, when we have a lot of indices, the problem would be the cluster state because as you handle a lot of index information, the cluster state gets very big.
22:21
So we needed to be multi-cluster and we built, some clusters where we put some store indices on any cluster and we replicated the common ones through all the clusters.
22:41
And we handle the search through clusters ourselves. Basically, we built a handmade router that has a table and when we ask for a store, it will route you to the right cluster. That's it. So it's a big infrastructure,
23:00
but it guarantees you, again, the quick search. To finish, some things you have to think about when it's about running. The first important thing would be to monitor what is happening on the search engine.
23:22
So we have to monitor what are the most frequent search terms, the filters. What happens? Well, what are the searches that gave zero results? What were the clicks on the second pages?
23:42
Where are also, when you search for product, where was it in the page? You also need histograms through the day. When is it most important? And the response times. It will learn you a lot about how it is used and this is also a part of scalability
24:02
because it's not only the fact that we have to grow a lot because it will always grow. We have to think that today we have thousands, we have two or three thousands of physical store. Maybe we'll have 10,000 in three years. We don't know.
24:21
Everything is growing, but also we know that the load is not the same throughout the day, throughout the different days of the week. So we also have to know about that in advance so that we be prepared when it's needed.
24:40
So this is just a quick view of what we're working on also. This is a proof of concept we did with pushing scalability to its best part. So we built our solutions. You see, if I get back to this one,
25:03
every part, most of this one is not dependent on the data. It is algorithms and stuff you have to proceed and this part can be seen as a microservice and we are able to scale.
25:20
So the idea would be to have different runners and we did this, we tried to do this using Kubernetes spots so that we can really scale up and down and see how we can with a bit of separation of concepts, be even thinner and more precise in the scalability.
25:48
So I told a bit, well, this was a quick view of what we have to think. Here's a checklist of what we just said. So first one, it's very important to keep adapting
26:02
to the place we are, the physical stores because the specificities are very important and physical stores always exist and it's important to stick to them. We can still use a schema or structure the data when we include all the sets coming from the web.
26:25
The response time is non-negotiable. This is really important. It's also important that your indexing is precise and also your data keeps coherent but response time is the most important.
26:40
It's important to monitor, to learn about what you monitor and to prepare for skilling even more. Try to think microservices for what can be scaled, what is not dependent on data. It can always be important
27:02
and I didn't speak a lot about security because to us a search engine is really in the backend. It's behind VPNs and stuff. So, well, security is important but we handle product data only, not personal data. So, well, the most important for us
27:22
is when we talk about security is to guarantee data integrity, error spreading, index corruptions that can also come from the inside of the system. So, that's it.
27:41
Thank you all. I think I can breathe now because I just came down south from upstairs so it was a bit complicated but thank you for your attention. Well, I'm open for any question you have. Have a great day. Thank you.
28:11
Thank you. You didn't mention testing of search relevance for example or where does that fit in? Is this part of your search console?
28:23
Like testing boost, search configuration. Just whether the results are relevant to the query. Well, we have different parts that include testing. First of it is code testing. So, obviously you need to check the integrity
28:43
when you do algorithms, you have to test them. Then you have also a big important part is the integrations testings. Also, when you are configuring very thinly with the search console, which is here. Yes, here you are doing very important things
29:05
that can be dangerous or can be corrupting also indices because you can also work on directly the elastic search settings and stuff. So, we have only manual integration testing at the moment with a pre-production environment in integration environments.
29:23
But we, well, in the configuration part it's difficult to automate integration testing. So, we do manual pre-testing of, you know, we click, we try such results and check it.
29:41
That's it. Yeah, thank you. I have a question about the very important response time that you mentioned. So, one second, is there any granularity to that? So, I mean, is it that everyone leaves at one second or maybe like half of them and something like that?
30:03
I don't see what you mean. So, one second, is it the time where everyone leaves or like there are people already start to leave at half a second or something like that? Well, I don't have the exact, you know, review of what is one second and half a second.
30:22
I don't know, if like, for me, it's more of a general thought, you know, when you're above a second, everyone leaves. Then about half a second is that when you say one second for me, it's the response time of the whole chain from the front end to, you know, getting the results
30:40
displayed and all that. So, in our team, you know, we are, well, in partnership with the front end team and the whole thing must not be more than one second, but the search part must keep even thinner because you have all the display and stuff. So, this is why we say for search,
31:01
it must be half a second. Here's one more. Yeah, hi, thanks for the talk. If I understand correctly, you, all right, yeah, if I understand correctly, then you're querying three indexes at the same time for each store.
31:21
So, the store index, the warehouse, and the. Yeah. How do you combine the candidates from the different indices? Combine and rank, so. Yeah, by chance, we have some common, you know, properties. Mainly the product identifier, the product card, you know, the main name, identifier, description,
31:43
more or less, is the same. So, we are able to get that and then get that back to the results. This is why I talked about the schema. You know, the common schema is important because you need to really have the, well, be able to combine information.
32:02
So, for the search terms, it will be such, you know, in any unitary specificity of a product in the specific index, but then when we retrieve the results, we will only be able to display a little part because what is common between the three indices.
32:21
You need to have a common structure, a bit of a common structure between the indices to do that. Okay, next question here. Yeah, hi. It's nice to have separate indexes, but I wanted to understand how do you organize your code? Do you have separate code bases for each store or is it the same code base
32:41
but organized in a certain structure because there may be differences between the stores that are very specific to the store. So, how do you manage that? You have the same code base. Really, we keep to keep the same one, but of course, the drawback of this
33:00
is that you have a huge configuration part. So, we are still with this currently. So, with one code base, a big configuration part and then you can fine-tune specificities and the configuration will be different regarding the index you are in. So, we can still do it with one code base.
33:22
I do not exclude that. Someday, if really you have different, you can say marketplace is a different business from a grocery store. So, it's not excluded that someday you would need to do plugins and stuff, very specific. We also have a version where we use plugins
33:43
and connectors specific for any stream also. So, there are different possibilities. Currently, we try to keep one code base, one product. Yeah, here.
34:03
Hi, thanks for the talk. It's very interesting, but I have a question similar to the one. So, you mentioned that you have so many indices, one per store, right? And how are you maintaining it? And when you scale, let's say you have thousands
34:20
of stores as your customers, then how do you maintain your settings? Let's say you have a separate setting on mapping for each of the index, then how are you maintaining the Elasticsearch updates? Let's say you have to change some mapping in an index. How are you maintaining this indices?
34:45
Well, it's a bit of the same answer. It's more with the configuration. So, you handle it with configuration. You have a very, in the store indices, you still have a lot of similarities. You have the same structure.
35:02
Not everything is different. The data is different. For the settings and mapping, I would say it's very similar. So, you can assume that your code will be the same, that you will be able to handle them on the code part the same way.
35:21
You see, you apply the boost, but the values will be different, but the keys will be the same. I don't know if it answers your question. Maybe not exactly. We can talk about it later if you want. Okay, maybe last question. Hi, thank you so much for your talk.
35:41
Pretty interesting point of view. I have a couple of questions. As you know, when you have a sales event, a Black Friday, there are a lot of events. Therefore, are you out of scale in Elasticsearch? Do you have any kind of recommendations or custom metric to do it?
36:03
So, until now, for the Black Friday, the infrastructure was handling it. We still don't have the Kubernetes thing. So, you know, it worked with the multi-cluster. Currently, the clusters we have
36:20
are not full. They are, you know, they can handle more, and it worked, it worked. We have, you know, five clusters currently with a lot of people requesting, and the load was maybe more in the indexing part, you know, all the corrections, all the everything that was happening
36:42
just before the Black Friday, all the ingestion part. And then the queries, as they are, they still keep very light, very unitary. Then, well, with, you know, only infrastructure, you can handle the load, currently. You know, with the,
37:01
it was the router also handling everything, but, well, it, we could do it. With Elasticsearch for that, I would say it's a bit of the same. You know, this is the scaling, the CEC also has a scalable thing. I think that Elasticsearch is also available,
37:22
for example, on Kubernetes, so you can, you know, scale, and unscale for such events. Okay. So thank you. Thank you.
Empfehlungen
Serie mit 2 Medien