The life of a search engine administrator
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 56 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/67203 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Berlin Buzzwords 20221 / 56
22
26
38
46
56
00:00
Musical ensembleInformation technology consultingXMLUMLLecture/Conference
00:33
Software developerEnterprise architectureSystem administratorContext awarenessKey (cryptography)Search engine (computing)Enterprise architectureSystem administratorVertex (graph theory)Musical ensembleProjective planeGoodness of fitMultiplication signProcess (computing)Context awarenessPhysical systemKey (cryptography)Lecture/ConferenceComputer animation
02:29
Linear regressionTime evolutionLevel (video gaming)Event horizonSearch engine (computing)Domain nameSoftwareLatent heatCASE <Informatik>Lecture/ConferenceComputer animation
03:14
Linear regressionTime evolutionLevel (video gaming)Event horizonEnterprise architecturePersonal digital assistantDifferent (Kate Ryan album)WebsiteInformationInformation securityCASE <Informatik>Context awarenessExpected valueCycle (graph theory)Product (business)BlogLevel (video gaming)Search engine (computing)WebsiteEnterprise architectureGoodness of fitHybrid computerTerm (mathematics)Matching (graph theory)System administratorDifferent (Kate Ryan album)Event horizonVideo gameComputer animation
06:39
Time domainInstance (computer science)CASE <Informatik>DatabaseContent (media)RootProcess (computing)RoutingOrder (biology)Lecture/ConferenceComputer animation
07:21
Content (media)Time domainGraph (mathematics)Different (Kate Ryan album)Subject indexingInstance (computer science)2 (number)Order (biology)InformationContent (media)Query languageMathematicsPhysical systemContext awarenessDomain nameProduct (business)Computer animation
09:06
Time domainGraph (mathematics)Group actionPressure volume diagramSubject indexingQuery languageOcean currentContext awarenessPrice indexPhysical systemSubject indexing1 (number)Multiplication signQueue (abstract data type)Search engine (computing)WebsiteLecture/ConferenceComputer animation
10:56
AverageRankingDecision tree learningCloud computingGroup actionWeb pageBit rateMeasured quantityDecision tree learningResultantProduct (business)NumberRankingAveragePrice indexInstance (computer science)Physical systemMeasurementShared memoryMetric systemLecture/Conference
12:32
Error messageSource codeTime domainSystem administratorOrder (biology)Search engine (computing)Error messageEnterprise architectureLatent heatProcess (computing)InternetworkingDomain nameWeb crawlerWebsiteGraph (mathematics)Term (mathematics)MathematicsTrailLecture/ConferenceComputer animation
15:11
Source codeSystem administratorProduct (business)Open sourceProcess (computing)SoftwareWritingPlanningChannel capacityPhysical systemSystem administratorMusical ensembleMappingSearch engine (computing)Projective planeLevel (video gaming)Process (computing)Lecture/ConferenceComputer animationProgram flowchart
16:05
System administratorProduct (business)Physical systemData managementEnterprise architectureProjective planeSearch engine (computing)System administratorLink (knot theory)Process (computing)Source codeSoftware developerOperator (mathematics)SoftwareSubsetProduct (business)Wave packetUniqueness quantificationComputer animation
18:30
SoftwareOpen setSearch engine (computing)System administratorLecture/ConferenceMeeting/Interview
19:06
Traffic reportingVideo game consoleSearch engine (computing)System administratorAnalytic setInformation engineeringGame controllerProcess (computing)Cycle (graph theory)Traffic reportingComputer animation
20:24
BlogVideo game consoleVideo game consoleCASE <Informatik>Extension (kinesiology)Service (economics)Core dumpMereologyComputer architectureSystem administratorMusical ensemblePoint (geometry)Analytic setAbstractionOrder (biology)Level (video gaming)Power (physics)Module (mathematics)Search engine (computing)Formal verificationWordResultantLoginConfiguration spaceQuery languageArrow of timeMappingSource codeDifferent (Kate Ryan album)Subject indexingSoftwareDomain nameWeb pageMathematicsLecture/ConferenceComputer animation
25:30
Search engine (computing)System administratorTask (computing)WebsiteCalculationMathematicsResultantLecture/ConferenceComputer animation
26:16
Video game consoleConfiguration spaceIntegerBuildingLinear regressionAnnulus (mathematics)Product (business)Instance (computer science)Electronic mailing listProduct (business)ResultantMathematicsEvoluteComputer animation
26:52
Formal languageHash functionSource codeMetadataDomain nameRootHypermediaContent (media)Web crawlerBuildingPhysical systemMathematicsProduct (business)Context awarenessEnterprise architectureSource codeProcess (computing)Field (computer science)File formatElasticity (physics)Search engine (computing)LoginSystem administratorData modelDifferent (Kate Ryan album)Lecture/ConferenceMeeting/InterviewComputer animation
28:32
Discrete element methodView (database)ModemSearch engine (computing)Physical systemPerformance appraisalTask (computing)Lecture/Conference
29:09
RankingData modelAlgorithmSoftware maintenanceFunction (mathematics)Query languageSearch engine (computing)System administratorMusical ensembleArithmetic meanCASE <Informatik>Interactive televisionAlgorithmSoftware testingWebsiteMereologyTask (computing)Computer animationLecture/Conference
31:00
Machine learningConfiguration spaceField (computer science)Web pageTime domainData modelVirtual machineNumerical analysisClassical physicsScale (map)RankingArtificial intelligenceSoftware testingAutomationPerspective (visual)Link (knot theory)Video gameCycle (graph theory)Search engine (computing)Performance appraisalOperator (mathematics)FreezingEvoluteSoftware testingProjective planeDiagramPhysical systemComputer animationLecture/ConferenceDiagram
32:29
Control flowFaktorenanalysePerformance appraisalTraffic reportingVideo gamePresentation of a groupSearch engine (computing)DivisorInformation engineeringBuildingProcess (computing)Focus (optics)Proxy serverGoodness of fitLecture/ConferenceComputer animation
33:56
Search engine (computing)EvoluteLecture/Conference
34:30
Musical ensembleSystem administratorChemical equationVideo game consoleConfiguration spaceLecture/Conference
35:04
BuildingComputer fileSearch engine (computing)Configuration spaceSoftwareSystem administratorIntegrated development environmentSoftware developerWave packetMappingLecture/Conference
36:18
MappingInstance (computer science)CASE <Informatik>Subject indexingMathematicsText editorFile formatType theoryField (computer science)Computer fileGame controllerLecture/Conference
37:11
FingerprintEnterprise architectureGoodness of fitContext awarenessBasis <Mathematik>Open sourceExpert systemConnected spacePhysical systemResultantDomain nameGradientGroup actionEvoluteMathematicsInformationBuildingClassical physicsLecture/Conference
39:38
MathematicsCASE <Informatik>Enterprise architectureCausalityMeasurementLecture/Conference
40:19
Mobile WebProduct (business)Enterprise architectureInformationRemote procedure callMultiplication signLecture/Conference
41:15
CuboidWeb pageProduct (business)FeedbackMusical ensembleResultantoutputSearch engine (computing)Lecture/Conference
42:01
Multiplication signMusical ensembleLecture/ConferenceJSONXMLUML
Transcript: English(auto-generated)
00:07
So, thank you for coming. We are from a company called Adelion, and we are search consultants at Adelion.
00:23
We are quite a few of us, so you can find us in the conference. We are doing a lot of consulting around search, and we are also developing a technology called A2, which is a vertical solution for e-commerce and enterprise search.
00:41
So, given this experience, we will kind of try to talk about what is to be a search engine administrator. So, first of all, why is this talk, what is it about? So, the main goal and the first idea of this talk was to look back on our jobs, our experiences,
01:10
so maybe kind of a mid-life crisis of a search engineer, I don't know, and to capitalize on what we've done, what we've learned, and so we imagine that the best way to do
01:24
that, because we usually don't take time to do this kind of look back, is to propose a talk, and hopefully it was accepted, so here is our talk.
01:41
So follow us on this path, so we will talk about the search engine as a living thing, so we see it as an ever-evolving entity. We talk about the context, in which context do we work, and are we interacting with.
02:05
Then we talk about the keys of success, what we see as the keys of success of a good search system. We focus on the different roles that we can have in the context of a search project, the tools that we use in general and we find very helpful, and the conclusion.
02:31
So we start kind of with a conclusion already, so the search engine is a living thing, search has the particularity that you cannot just throw a software there, install it, and it
02:47
keeps working like this alone, you have to constantly improve it. For different reasons, depending on the domain in which you are in, you have seasonality,
03:02
for example if you are working in e-commerce, especially for example in the case, specific case of food, you have a lot of seasonality there, you also have, also in e-commerce you have the level of stocks, you have the life cycle of different products or the life
03:22
cycle of your documents, and you have also events that happen during the year, so you have to adjust your search engine, you have things that come like in the news, and you have to adapt, just like you adapt a website, and you also have the users that
03:44
evolve themselves, so their expectations change. So you have either two paths from here, either you just leave the search engine as you installed it and as you configured it in the beginning, or you can adapt it
04:02
constantly so the search engine evolves. So the context, there are different contexts in which we are placing the search engine administrator,
04:20
and depending on the context, the life of the search engine administrator is not the same, and also the skills, the people around the search engine administrator are not the same. You have a first use case which is really specific, which is the e-commerce use case.
04:42
In this case, the relevance for example is not in terms of text and what matches and what does not match is a text, the relevance is in terms of stocks, marketing, agenda, events, personalization also because in e-commerce you have to really take into
05:08
account what the user wants and where the user is located and stuff like that. A different use case is the enterprise search use case where the goal of the search engine
05:27
is that you are not selling goods and the search engine is configured based on different criteria. As the search engine administrator, your KPIs are mostly connected to the productivity
05:42
of your colleagues or users that are using your search engine. You also have an issue with the security because in this context, which is enterprise search, documents are really sensitive, so you have to take care which has access to which document.
06:04
And then you have some hybrid search engines. One straightforward example is where you have an e-commerce search engine that sells goods and also is meant to be a reference website. Reference website means that it also has blog posts, technical documentation, stuff like that,
06:29
which are there to make the user come back to your website and to consider it as a reference website and not only an e-commerce website. You also have a specialized use case that does not really fit in one of these use cases.
06:46
Like, for instance, scientific databases or legal content, which is generally high value content, and the search is really critical for professionals to leverage this content.
07:04
We did not really focus on it, but it's also a use case. So, going back to the root and the guys of our job, what is search about? So, we think that basically search is about leveraging signals in order to fulfill a user's need,
07:29
whether it's a need for information, for a product, for anything that it can be in search for.
07:42
And we did moving signals. So, as Lucian said a few seconds ago, the search is operating in a moving context. So, the signals that we are using to fulfill this need and to build the search are moving.
08:04
So, for instance here, we try to show and to make visible the different signals that we use to build a search system. So, usually, and maybe in the first place, when you start a search system, you work with a query and the content.
08:27
So, first you have the content, you index the content, and then you try to have a query which matches this content, and you have the query and you make them match. But you can have more, you can have user knowledge and domain knowledge, and these things usually make the difference.
08:47
And so, here we have the different topics linked to users, domain, content, and query that we usually leverage to build the search system.
09:03
You also have to take into account that the data changes. So, even if your query is correct today on the current data, tomorrow the data will change and the query might not be correct. Yes, so we keep an eye on all those entities and things.
09:21
So, once we have our context and our data, let's say, in the general sense, we must define what is success, what is to fulfill a user's need. And so, here we are in what we call the KPIs jungle. We have a lot of possible indicators to indicate that search is successful.
09:43
First, we have the user's indicators. What do users say about your search system? So, in e-commerce, we often have NPS, not always specific to a search engine or to the search system of your e-commerce site, but it can be a good indicator.
10:02
You have, of course, business indicators, so it's a way to see if your search or your e-commerce system generally is working well. You have, on the other side, technical indicators, search time, indexing queue size, uptime of your system.
10:27
You have also relevance indicators, so these indicators are also very useful, but in general, how will you use them, which are the good ones to use.
10:40
So, in general, the most used is the last one, famous LGTM, so it looks good to me. It's the indicator that we use when we do not have a better one. It looks good to me, so it's okay. So, here is the way we process, in general, to organize these indicators and
11:08
formalize them and have a better way to define the success of the search system. So, we start with the goal, the center of the target, and with the business, we try to expand this to actionable metrics or KPIs.
11:33
So, further success can be increase the sales, but also maybe increase the profit or the market share. It can lead to different indicators, and then we extend it to measurable things, measurable quantities linked to the search.
11:50
So, for instance, average rank of the product when it's added to the cart, the number of clicks on a search result page to cart, the number of zero result search, etc.
12:05
One which is very interesting is the rephrase rate when a user does a search, and the next action is to rephrase a search because obviously it did not yield satisfying results.
12:20
So, all those are the measurable KPIs, and we process like this, starting by defining success with, generally, the business owners. So, in order for the search engine administrator to be relevant in his work, you
12:45
need to monitor, you need to monitor a lot, you need to know what's happening. And the first question that arises is what to monitor, how do you keep track of how the search engine is used.
13:03
So, first of all, you have to monitor the relevance of your search engine, so this is defined in terms of like business needs. And you also have some other monitoring tools and graphs that you need.
13:21
For example, when you're talking about an enterprise search engine, your search engine might not be relevant because of crawling errors. Because there's an intranet that's not accessible to the search engine, and actually your data is not there, or your data is there, but it's not complete.
13:43
You have to monitor the errors on data because you might have access to the data, but the data is corrupted or is simply not appropriate to be indexed into a search engine as it is. So, you have to first like take care of the data, to clean it, to improve it, to extend it.
14:07
And also, you have to keep an eye on the user satisfaction. So, user being like your colleagues, if it's an enterprise search or the customers of your e-commerce website.
14:21
So, as a search engine administrator, you have to keep an eye on kind of all of these things. So, business, data and infrastructure, and end users. So, we mentioned the KPIs, some of them. So, another change is how to define the KPIs.
14:45
So, it's also the job of the search engine administrator to like be the one that suggests which KPIs should be put in place. And the KPIs are important. You also have domain specific issues and you have to handle them. So, it's things that are related really to your business.
15:10
So, quite a lot of challenges and we try to organize them to focus on the different roles of the search administrator.
15:21
And so, as you see, I like the mind maps. So, I'll draw another one. So, at the center, so the search administrator or search engine or specialist or colleague as you want, we find this way to call our job, so search administrator.
15:44
So, we have a level of relationships with all the stakeholders in general with the projects we deal with. So, in e-commerce, we have the voice of the customer, the teams of the customer, which are usually connected more formally in e-commerce.
16:06
Can be collected also in enterprise search, besides the coffee machine, somewhere like this. You have the business owners in general. So, with the business owners, we have a lot of education to do in general.
16:24
Explain how it works, the challenges, define the KPIs we talked about earlier. Design some tools. We'll talk about just after the tools to empower the stakeholders. With the product owners, we do more project management stuff.
16:46
So, UX design, search story, et cetera, acceptance criteria also. Then we interact because we are IT specialist, so we interact with IT. So, IT ops, which can be also not only operators of search, but users of search because a lot of monitoring tools use also internally search engines.
17:11
So, we also sometimes have a double role in the project, one for the business, such as a business tool and as an ops tool.
17:22
We interact a lot, of course, with the IT dev team for trainings, for software design. We also contribute to the development. We interact with the data team. So, in general, you have data teams that collect as much data as possible.
17:41
So, we feed the lake. We contribute to future engineering because we can help understand the data that are collected, et cetera. We also use back the data to collect signals, which will also help us doing our job of relevance engineers.
18:04
And, of course, we interact, so I put it maybe a little further away, with knowledge sources because we have to also adapt to knowledge. So, it's not really related to our job inside the project, but it's something
18:21
that we must always keep in mind to keep a link with increasing our knowledge. So, the search engine administrator also helps defining the roadmap. So, if your search engine is not yet the ideal one, it's also the role of the search engine administrator to figure out what's next, what's to be developed next.
18:44
And, yeah, to define and prioritize the backlog. Okay, so, as I just said, one of our roles is to empower the stakeholders to use some tools.
19:09
We don't want to do everything by hand. We don't want every team to rely on our job. So, our role is also to design with the stakeholders what tools can help
19:23
them better do their job, have quicker release cycles, faster go-to-market, et cetera. So, empower first the business people. I put merchandising, so we talk about merchandising nowadays also.
19:44
So, give the controls to adjust the way the search works for business people and not for technical people. Empower the reporting teams, so giving them analytics tools, dashboards, and also empower the data teams,
20:03
as I said, to automatically export some data and help them do their data engine interrupts, basically. So, tools that we build or we also use existing tools, of course.
20:23
So, there's this tool that we call Business Console, which is kind of the ultimate user experience for a search engine administrator. It's not necessarily a single tool, so you can have it, like, usually you have plenty of tools that are there.
20:43
But if you manage to bring them together to have just one entry point, which is this Business Console, which is the wheel that the search engine administrator is using to control everything, then it's fine.
21:07
So, as a search engine administrator, you have to, like, keep an eye on all the three parts of a search engine. So, this is actually taken from our architecture, from our software, and you have the ingest part,
21:21
which is, like, crawling, transforming the data, gathering the data, and ultimately indexing it into your search engine. You have the search engine technology, the core search engine technology itself, which needs to be configured, which needs to be integrated and all that, and you also have the search services.
21:41
And to a certain extent, you also have the UI and the connectors to different data sources. So, in any case, a search engine administrator has to keep an eye on all these parts of a search engine, and we believe that all these are part of a search engine.
22:03
And if you can provide the search engine administrator a tool, a centralized tool that keeps an eye on all these things, then that's the best user experience. If I go into details, so, what could you offer to a search engine administrator so it has most of the tools that he needs?
22:28
This is an example from a search engine based on Elasticsearch, but it's the same with Solr or other technologies. So, you need something that allows the configuration of the core search engine itself. So, that's the configure module.
22:48
So, these tools here should be technical enough so you have a lot of power, but also high level in order to, like, to make abstraction of the technology that's behind that.
23:06
Then you also need access to logs. All the logs you can get. So, this actually logs module needs to have access to indexing, to the search services, to your UI logs, analytics and all that.
23:25
Then the pilot module is the one that empowers the search engine administrator to act, to do things, to, like, for example, relaunch a crawling, relaunch an indexing, reindex some data, clean some data, change the mapping, change a query, change a search query and all that.
23:52
Yes. So, yeah, actually there's an arrow missing there because the pilot can also act on the queries. You also have to have a debug module which is, which empowers the search engine administrator to know what's happening in the search engine.
24:09
Like, when you see a result on the front page, you have the details that tell
24:21
you why that result is first and that result is second. So, this is the debug module. And then you have a lot of what we call helpers. For example, when a search engine administrator defines synonyms, okay, either they know the domain and they can define the synonyms
24:43
because they know the business, but you also can provide helpers that will actually detect automatically synonyms. There was a talk or there will be a talk about that in this conference about how to automatically detect the synonyms. We are not saying that you have to automate the synonym definition.
25:06
It's still the search engine administrator that has the last word and has to do the verification. But this tool can help. You also have some non-regression tools. There has been a talk by people
25:21
from Auto that presented some tool like this. Yeah, you had NDCG or all that. Okay, so just an example to illustrate a tool that we built with our colleagues for search engine administrative tasks.
25:45
So, it's an example in an e-commerce website. So, the administrator, the business people can do a search, run a search, see the result as a customer would see, plus a score explanation here.
26:03
So, why the score computed by the search engine is like this? So, it's kind of mathematical, but you can have very detailed explanation on the score calculations. Then you can act on the way the search is run. So, by, for instance, selecting some facets, moving the facets, et cetera.
26:29
And then after committing the change, you save the change that you did, you can reproduce the search. So, search again and see the evolution of the result list. For instance, this product is two places higher in the result list. This one went down.
26:52
So, you can really evaluate the impact of your changes on the search system before putting it in production.
27:02
And this particular tool reinforces the looks good to me. In the context of enterprise search, but not only it can also apply to e-commerce, you have sometimes a lot of heterogeneous data.
27:25
And the tool that you can provide to your search engine administrator is like a data modeling tool. Which is not really like something that's well defined. You have, for example, people from Elastic that built things that call Elastic Common Schema.
27:44
So, this was for them to empower those that use data coming from logs to have the same format. So, the administrator will just say, oh, I have a new data source here. My job is just to map fields from this new data source to the common schema that's the centralized one.
28:06
So, when your search engine uses a lot of heterogeneous data and data coming from different data sources, you can empower him with a schema, for example, that would be like the ideal schema of the data.
28:22
And when a new data source arrives, you provide a tool that allows you to go into this. So, we saw a lot of tools, a way to collect data, tools that are basically manual tools.
28:43
Not the task of the search engineers, but the business users. They must always watch how the search is performing, do the adjustment, assess the impact of the adjustments. But maybe we could, with all this data and all these evaluations we collect, can we have some
29:06
kind of magic AI system which could crush all the data and make the search engine adapt automatically? So, you could ask, is AI the end of the search engine administrator? Because you can
29:22
just plug AI and then the search engine will configure itself and evolve by its own. Well, it helps, but we think that it's not really the case. This is an example from another talk that we gave, and that is the conclusion of what we learned when implementing learning to rank.
29:50
Which is somehow supposed to automate some administrative tasks there and to adapt to the data, to adapt to the users and all that.
30:01
So, I will not get into details, but the conclusions were that the part that was done manually, and we actually did this test for a large e-commerce website, and we kind of like moved back from learning to rank to like a manual or semi-manual administration.
30:27
Because a search engine administrator that knows the business and knows how the users interact with the search engine eventually behaves better than any AI algorithm.
30:44
You also have, yeah, so you have Trey Granger that used to be a speaker in this conference that said that all this AI stuff is kind of a buzzword salad, so it will not really replace the search engine administrator.
31:01
And if you want to know a little more about this, there's a talk that we gave last year at Buzzwords. Of course it was not in real life, it was a remote conference. You can see more about our opinion about AI.
31:20
Okay. And another thing to also go further in the topic is a conference that Peter Friis gave at Haystack in 2018. So we like this kind of cycle of the search operation, so how we make search adapt, so in general we evaluate,
31:48
and then we run the lab wheel, so we do offline relevant evaluations, improvements, automatic testing, etc. until we have a satisfying lab cycle, and then we go to the operation cycle.
32:07
So we cited this diagram because we found it very interesting on how to represent the evolution of a search system applied to a project.
32:23
And we get back to the fact that the search engine is a living thing, so it has to constantly evolve being in the lab or in the real life. So quick conclusion because we are reaching the end of this presentation. So what we believe are the success factors of a search engine.
32:50
Define your KPIs first. Know what is a good search. It's often not really easy because the relevance is usually in the eye of the builder, of the searcher.
33:06
So everyone has his own definition, so find some KPIs, try to find some proxy KPIs to evaluate what is a successful search. It's not always easy. Empower your stakeholders, give them tools, use tools, build tools, collect as much data as you can,
33:27
even if you don't know why in the first place you will hopefully have data engineers or data scientists which can help you use those data to improve the signals that will be your prime matter to work the search relevance.
33:49
And so a little focus on the tools to remind what tools are really useful in our jobs and our customers' job. So at the end, be the search gardener. We say that for us the search engine is a living thing. We compare it to a garden.
34:07
So the evolution of the garden is up to you. Will it be a wild garden, a wild life, or will it be fruitful? It's up to you.
34:24
Okay. Thank you for your attention. Anybody has a question?
34:47
Thanks for the talk. How do you balance allowing your admins to modify things manually in your console versus having declarative configurations?
35:00
So the question was how do we balance the manual configuration with? Having a declarative file where you can put it on Git or it's like a more declarative approach versus just manually. A more automated approach. I don't know. You have to have both actually.
35:28
And the tools that you give to your search engine administrator can be technical enough and you can also do some training, for example. So the search engine administrator actually uses Git and actually if the
35:47
software that you provide, for example, allows you to export a configuration, then you can even empower your search engine administrator to push it to Git and to have a semi-developer approach there.
36:05
So yeah, it's kind of a hybrid approach. You have to use both and to adapt to your environment. Just to complete, for instance, if you consider a mapping in Elasticsearch, in our
36:25
case the user, the business user, can really modify the mapping, the real mapping. So it modifies by changing maybe the field type or data format, date format, things like that.
36:40
And the mapping is changed. It's committed with a new change. It's applied on the index and the user knows that when the mapping changes it has to re-index. So it will click on re-index. So the real mapping will be changed. And after it's up to us to save it in Git or somewhere. But it gives a real control on the real file that usually we edit on a text editor.
37:15
Thank you for the talk at first. Do you have any good practices or references for finding KPIs in an enterprise search context?
37:30
In enterprise search it's usually kind of difficult to find good KPIs. We
37:41
have, because here also the needs can be very different, the information needs. So we can, in enterprise search, we often tend to recommend expert judgments. So we try to have some domain experts run some search.
38:04
So maybe we take the top 100 searches or top 200, give them to the domain expert in the company. And we tell them, okay, run the search and give a relevance grade to each result.
38:23
So there are some tools. There is a tool called Cupid by Open Source Connections. It's an open source tool that can help collecting relevance judgments and expert judgments. And it will help you build relevance basis of truth that you can use to evaluate your search.
38:48
So when you do some changes, you add some documents, you add a new, I don't know, you add new users, you have an evolution in your search system, you can evaluate if the search is still performing well.
39:03
So usually a lot of, let's say, domain system experts, domain subject experts, you also have the typical classic KPIs like zero-result search, search rephrasing.
39:25
So the next action is to run a new search. But it's not very specific to enterprise search, but it's useful in general in all search systems.
39:42
What I'm observing is that in the enterprise, in the end, you need a business case. And talking about KPIs changes the movement from talking about costs to talking about use.
40:01
This is a great movement because in an enterprise, always costs are the first thing to talk about. And if you have KPIs, you must translate those into something measurable like euros or dollars. And especially if you're talking about enterprise search, it's really hard to figure out how much minutes do you save using research.
40:33
Yes, and we know every enterprise search vendor, okay, with our product productivity increased
40:41
by 50% or typical knowledge worker spends 20% of time searching for information. Yes, it's difficult to show the return on investment, but something when we are, especially with remote
41:04
work today, I think enterprise search becomes more than a commodity, something really mandatory for every company. It ends up to things related to user satisfaction. If you have a little box there that will ask
41:26
input from users on a search results page, and users give a good feedback, and the user experience is better.
41:40
So yeah, the KPI will be something related to the happiness of the employees, for example, for those that use the search engine. And also to the productivity, for example, but it is difficult to measure it, but you have to find something connected to that. Yeah, so happiness and productivity.
42:02
Okay, so unfortunately we ran out of time. Thank you Lucian and Vincent. Thanks for the talk. Have a warm applause, please.
Recommendations
Series of 2 media