Building a distributed search engine with tantivy
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 542 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/61634 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Distribution (mathematics)BuildingExecution unitSeries (mathematics)Read-only memoryType theoryQuery languageMultiplication signCuboidBefehlsprozessorState of matterGoodness of fitWordResultantCountingSemiconductor memoryBitPartial derivativeQuicksortWebsiteLoginSubject indexingSearch engine (computing)MehrplatzsystemHigh availabilityPoint (geometry)MiniDiscThread (computing)Software developerQueue (abstract data type)Resource allocationNumberPhysical systemMetropolitan area networkField (computer science)Category of beingSystem programmingFitness functionDistanceLimit (category theory)Form (programming)Data structureCore dumpImplementationOrder (biology)CodecRange (statistics)
08:47
Dynamic random-access memoryComputer wormRaw image formatProcess (computing)ConsistencyTrigonometrySystem programmingData centerQuicksortClient (computing)Meta elementDiagramField (computer science)Data storage deviceSoftware testingGodCore dumpPeer-to-peerData compressionMiniDiscBitOperator (mathematics)WordOpen setAlgorithmHigh availabilityImplementationIndependence (probability theory)Subject indexingCASE <Informatik>Complete metric spaceBlogLevel (video gaming)Library (computing)Bounded variationBlock (periodic table)BefehlsprozessorConsistencyComputer fileState of matterFinite-state machinePhysical systemServer (computing)SoftwareTerm (mathematics)Arithmetic meanProcess (computing)Matching (graph theory)Enterprise architectureShared memoryQuery languageFuzzy logicPoint (geometry)Inverter (logic gate)Regular graphSoftware frameworkWebsiteRewritingSystem programmingData management1 (number)Demo (music)Scaling (geometry)File archiverMereologyMultiplication signVideo game
17:28
System programmingDatenverknüpfungLocal GroupExtension (kinesiology)Unit testingData typeMultiplication signState of matterRun time (program lifecycle phase)Subject indexingSoftwareSoftware testingVideo gameComputer virusBitData storage devicePeer-to-peerKey (cryptography)Address spaceQuicksortSound effectString (computer science)Axiom of choiceLink (knot theory)Revision controlCausalityPoint (geometry)Physical systemSystem programmingResultantSemiconductor memoryImplementationForm (programming)Level (video gaming)Operator (mathematics)DistanceReplication (computing)Demo (music)ConsistencyDatabaseScaling (geometry)Vector space modelAlgorithmLibrary (computing)Element (mathematics)Right angleStandard deviationBuffer solutionLetterpress printingPlanningDatabase transactionMessage passingMathematicsVector space
26:09
Query languageDemo (music)ConsistencyRevision controlFreewareEinbettung <Mathematik>Existential quantificationWordData storage deviceHigh availabilityServer (computing)Finite-state machineBitMultiplication signRing (mathematics)Point (geometry)Virtual machineClique-widthImplementationSoftwareReplication (computing)System programmingSoftware testingAlgorithmQuery languageDivisorHash functionDistanceData dictionarySubject indexingMechanism designLogicSoftware frameworkLevel (video gaming)Latent heatMereologyConnected spaceOrder (biology)Vector spaceGroup actionAdditionState of matterComputer virusMathematicsMessage passingFlow separationDiagramDependent and independent variablesLine (geometry)
33:42
Program flowchart
Transcript: English(auto-generated)
00:05
I'll just yell, I'll yell. But yeah, so this is effectively my talk. It started out as a really big thing, and then I realized 40 minutes wasn't actually that much time. And so we sort of had to compress it down into a bit of a slightly smaller talk,
00:21
but hopefully covering the most interesting points, in my opinion. So a bit about me. I'm Harrison. I come from London. I live in London. And I work for Kwikwit, where, as Paul has said, we build basically a distributed search engine for logs. I am the creator of LNX, which is a slightly different design
00:44
of search engine, probably more akin to something like Elasticsearch or Algolia for all your lovely e-commerce websites. And you can contact me at Harrison at Kwikwit.io. A little bit about LNX, since this is basically
01:01
the origin story of this talk, really. It's a search engine built on top of Tantv. It's akin to Elasticsearch or Algolia, as I've said. It's aimed at user-facing search. That's things like your e-commerce websites, your Netflix streaming platforms, things like that. It's not aimed to be your cost effective log search engine.
01:22
It doesn't really handle those hundreds of terabytes a day type workloads, but it will handle thousands of queries a second per core. It's very easily configurable. It's designed to be really fast out of the box because it uses Tantv. And it has an indexing throughput of about 30 to 60 megabytes a second on reasonable hardware,
01:42
with high availability coming soon, which is the presence of this talk. So well, what is user-facing search? And I've stolen Crunchyroll's website, and I've typed some bad spelling in there. And you see that a lot of the top results actually account for the fact that I can't spell. And that's basically the biggest principle
02:01
with these user-facing search engines, is you have this concept of typo tolerance. This is a really good thing for users because users can't spell. The downside of this is that it adds a lot of CPU time when we're checking those additional words, and it makes things a lot more complicated.
02:20
And often documents are mutable and a lot of other things. But also, when you have these nice search experiences and you want low latency, something called search as you type has become more popular now. And that means your amount of searches you're doing for a single user is increasing several times over,
02:40
because now every keystroke you press is a search versus typing it all in one go, hitting Enter. User gets a bunch of results back, goes, oh no, I've spelt something wrong. I can't see what I want on here, so I'm going to type it again. And so that is effectively the principle of these search engines.
03:00
You see, we have Algolia at the bottom, which is a very common one, which I think most people know, very popular for document searching. But we decided, hey, we don't want to use one of these pre-built systems. We don't want to use Elasticsearch. That's big, that's scary, I don't like it. We don't want to use Algolia because I don't have that much money. I'm just a lowly paid software developer.
03:22
I can't be spending thousands of pounds on that. And we look at some of the others, but we go, no, we're just going to write it ourselves. And then that's where we have a little look, because we hear something about Tantivy, we hear something about Rust and it being blazingly fast, as all things must be, and so we go, okay, I like this. I like what it says.
03:40
It says, yeah, Apache Lucene, I think I've heard that before somewhere, written in Rust. I think I've definitely heard that before. And so we take a little look at what it is, and it is effectively akin to Lucene, which if you don't know what that is, it's a full text search engine as it's called. Tantivy in particular supports things like BM25 scoring,
04:01
which is just a fancy way of saying what words are relevant to this query. It supports something called incremental indexing, which basically just means you don't have to re-index all of your documents every time you change one thing. You have faceted search, you have range queries, and we have things like JSON fields, which allow for a schemaless indexing as such.
04:22
You can do aggregations, which have some limitations, in particular around the JSON fields being a little bit limited. But the biggest thing is it has a cheesy logo with a horse, which I believe Paul drew himself, so I think that that needs a clap on its own. But there are other features which I just haven't, yes.
04:44
But there are more features which I couldn't fit on this slide, and time is of the essence. So you might be wondering what the basic implementation of Tantivy looks like, and because it's a library, it's actually really quite simple to do. So we have a couple of core things starting at the top
05:01
as we define what's called our schema. Since Tantivy was originally a schema-based system, and still is, we need some way of telling Tantivy what the structure of our documents are, and defining what properties they have. We can use something like a JSON field to give the impression of a schemaless index,
05:21
but schemas are good, we should use them. They come with lots of nice bells and whistles, so in this case, we've created a schema with a title field. And you can see there we've added the text and stored flag, which all that really says is, I'm going to tokenize this field, and then I'm going to store it
05:41
so we can retrieve it later on once we've done the search. The second thing we do once we've done that, is we create our index writer, and in this case, we're just letting Tantivy select the number of threads, so by default, sorry. When you create this index writer, and we give it a memory buffer,
06:00
in this case, about 50 megabytes, Tantivy will allocate n number of threads, I think up to eight threads, depending on what your system is using. So you don't really have to put much thought into the multi-threaded indexing. And then we're just adding a document, really. So we've created our document, we've added the text field,
06:20
we've given it, in this case, the old man of the sea, and we're going to put it to our indexer, which is essentially just adding it to a queue for the threads to pull off, process, spit out onto disk. And then if we want to actually have that be visible to our users for searching and things like that, we need to commit the index. So in Tantivy, you can either commit or you can roll back,
06:43
and if you have a power failure midway through indexing, when you reload from disk, it'll be at the point of that last commit, which is very, very useful, so you don't leave with partial state and all that, all that nasty things. And then once we've done that, we can actually search. And in this case, you can either build queries
07:02
using traits, which are very nice, and you can mash them all together with lots of boxing and things, or you can use the query parser, which basically parses a nice little query language. In this case, we've got a very simple phrase query, as it's called, trouble that up, and it spits out a query for us.
07:21
We then pass that into our search executor, which in this case, we're executing the query, and then we're passing what are called collectors, and they are effectively just a simple thing to process the documents which have matched. So in this case, I believe we've got the count collector and the top docs collector,
07:40
and the count collector does, well, it counts, a big surprise there. And we have the top docs, which collects the top K documents up to a given limit. So in this case, we've selected 10. We only have one document to match, so this doesn't matter that much, but if you have more, you can limit your results, you can adjust how things are scored, et cetera.
08:02
Now, that's all well and good in this example, but this doesn't actually really account for spelling, and as we discussed earlier, users aren't very good at spelling, or at least I'm not. So we maybe, we want a bit of typo tolerance, and in this case, Tantry does provide us with some additional way of doing this in the form of the fuzzy term query.
08:23
It uses something called leaving signs distance. It's a very common form of effectively working out how much modification you need to do to a word in order to actually get it to match, and we call that the edit distance as such. Typically, you're between one and two edits,
08:41
so you're swapping a word around, you're removing it, you're adding a new word, a bit of magic there, really. And as you can see at the bottom, this is effectively, if we use just the regular full text search, if we enter the term hello, we'll only match with the word hello, but if we go with the term hell,
09:01
we'll only match with the word hell. If we use some fuzzy term query here, we can actually match hell and hello, which is very useful, especially for the prefix search. This is built upon Tantry's inverted index, which uses something called a FST, which is effectively a fancy word for saying,
09:22
we threw state machines at it, and then made them return results. That's as much as I can describe how they work. The person who originally wrote the FST library in Rust, Burnt Sushi, he has a blog on this, goes into a lot of depth, really, really useful for that sort of thing, but I can't elaborate any more on that.
09:43
But all of this additional walking through our index and matching these additional words does come at the cost of some additional CPU. And once we've sort of got that, what we're left with is this nice block of data on our disks, really. So we have some metadata files here,
10:01
in particular, meta.json, that contains your schema, along with a couple other things. And we have our sort of core files, which look very similar, if they look very similar to these scenes, that's because they are. In particular, we have our field norms, our terms, our store, which is effectively a row-level store log file,
10:22
our positions, our IDs, and our fast fields. And fast fields are effectively fast, because we cut, somewhat simple and equally a vague name. But now that we've got all this stuff on disk, if we wrap it up in an API,
10:42
we sort of, we've got that, we've got everything. In this case, we've got a demo of LNX working here, and we've got about, I think, 27 million documents, and we're searching it with about millisecond latency. I think in total it's about 20 gigabytes on disk compressed. Which are pretty nice,
11:01
but there's sort of a bit of an issue here, which is, if we deploy this production, and our site is very nice, we get lots of traffic, things increase, we go, hmm, well, search traffic's increased, our server's not coping, let's just scale up the server. And we can repeat this for quite a lot, and in fact, things like AWS
11:21
allow you a stupid amount of cores, and things like that, which you can scale up very easily. But you keep going along with this, and eventually something happens, and in this case, your data center's burned down. If anyone remembers this, this happened in 2021. OVH basically caught fire, and that was an end of, I think,
11:42
a lot of sleeping people. And so, yeah, your data center's on fire, search isn't able to do anything, you're losing money, no one's buying anything, management's breathing down your neck for a fix, you're having to load from a backup, what are you gonna do? And well, you think, ah, I should've made some replicas,
12:01
I should've done something called high availability. And in this case, what this means is, instead of having one node on one server ready to burn down, we have three nodes available to burn down at any point in time. And in this case, we hope that we put them in different what are called availability zones, which mean, hey, if one data center burns down,
12:21
there's a very small likelihood, or at least there's a possible for another data center to burn down in the meantime. And this allows us to effectively operate even though one server is currently on fire or lost to the ether or, I don't know, a network has torn itself to pieces. And this does also mean we can upgrade. If we wanna tear a server down
12:41
and we wanna restart it with some newer hardware, we can do that without interrupting our existing system. But this is sort of a hard thing to do because now we've got to work out a way of getting the same documents across all of our nodes. In this case, it's sort of a share nothing architecture. This is done by Elasticsearch and basically most systems.
13:03
So we're just replicating the documents. We're not replicating all of that process data we've just done. We need to apply them to each node and doing this approach makes it a bit simpler. In reality, LNX and QuickWit do something a little bit different, but this is easier. I say this is easier because the initial solution
13:22
would be, you know, just spin up more nodes. You know, add some RPC in there, what can go wrong? And then deep down you work out, it's like, oh, what do you mean networks aren't reliable? What's a raft and things like that. And so at that point you go, okay, this is harder than I thought. And you realize the world is in fact a scary place
13:42
outside your happy little data center and you need some way of organizing states independent on things catching on fire. And this is a hard problem to solve. And so you have a little look around and you go, well, Rust is quite a new system. It's quite a young ecosystem. We're quite limited, so we can't necessarily
14:02
pick a Paxos implementation off the shelf. We maybe have something called Raft. So that's a leader-based approach. And that means we elect a leader and we say, okay, leader, tell us what to do. And it will say, okay, you handle these documents, go do things with them. It's a very well-known algorithm, very easy to understand.
14:23
It's probably the only algorithm which is really implemented widely in Rust. So there's two implementations, one of them by the Pincap Group called Raft RS and the other by DataFuse Labs called Open Raft. Varying levels of completion or pre-made.
14:41
So in this case you think, okay, I don't really know what I'm doing here, so maybe I shouldn't be managing my own Raft cluster. And you hear something about eventual consistency. And you hear, oh, it's leaderless. Any node can handle the writes and then ship it off to the other nodes, as long as the operations are idempotent. And that's a very key point,
15:00
which means you can basically ship the same document over and over and over again, and they're not going to duplicate themselves, or at least they don't act like they duplicate. And this gives us realistically a bit more freedom. If we want to change, we can change. And so we decide, let's go with eventual consistency,
15:21
because, yeah, I like an easy life, and it seemed easier. Yes, people laughing will agree that, yes, things that seem easier probably aren't. And so our diagram sort of looks something like this. And I'm scared to cross the white line, so I'll try and point, but we have step one,
15:41
a client sends the documents to any node that doesn't really care which one. That client then goes, okay, I'm gonna send it to some of my peers and then wait for them to tell me that they've got the document, it's safe. And then once we've got the majority, which is a very common approach in these systems, we can tell the client, okay, your document is safe.
16:01
Even if OHV burns down again, we're probably going to be okay. It doesn't need to wait for all of the nodes to respond, because otherwise you're not really highly available, because if one node goes down, you can't progress. And so this system is pretty good. There's just one small problem, which is how in God's name do you do this?
16:22
Many questions need to be answered. Many things, how do you test this, or who's going to have the time to do this? And well, luckily, someone, aka me, spent the better part of six months of their free time dealing with this. And so I made a library. And in this case, it's called Data Cake.
16:41
Woo, yes. In this case, this is called Data Cake. I originally was going to call it Data Lake, but unfortunately, that already exists. So we added cake at the end and called it a day. It is effectively a tooling to create your own distributed systems. It doesn't have to be eventually consistent, but it just is designed to make your life a lot easier.
17:02
And it only took about six rewrites to get it to the stage that it is. Because, yeah, things are hard, and trying to work out what you want to do with something like that is awkward. But some of the features it includes is it includes the zero-copy RPC framework. And this is built upon the popular archive framework,
17:22
which is really, really useful if you're shipping a lot of data, because you don't actually have to deserialize and allocate everything all over again. You can just treat an initial buffer as if it's the data, which if that sounds wildly unsafe, it is. But there's a lot of tests, and I didn't write it, so you're safe.
17:41
We also add the membership and failure detection, and this is done using Chit Chat, which is a library we made at Quick Quit. It uses the same algorithm as something like Cassandra or DynamoDB. And this allows the system to essentially work out what nodes are actually its friends and what it can do.
18:02
And in this case, we've also implemented an eventually consistent store in the form of a key value system, which only requires one trait to implement. And the reason why I went with this is because if you implement anything more than one trait, people seem to turn off, and frankly, I did when I looked at the Raft implementations.
18:22
So we went with one storage trait that's all you need to get this to work. We also have some pre-built implementations. I particularly like abusing SQLite, so there is an SQLite implementation and a memory version. And it also gives you some CRDTs, which are conflict-free replicated data types,
18:42
I should say, and also something called a hybrid logical clock, which means it's a clock which you can have across your cluster where the nodes will stabilize themselves and prevent you from effectively having to deal with this concept of causality. And causality is definitely the biggest issue you will ever run into with distributed systems
19:02
because time is suddenly not reliable. And so we go back to our original thing of, well, first we actually need a cluster. In this case, it's really simple to do. All we need to do is we just create a node builder. We tell Data Cake, okay, we've got your address is this,
19:20
your peers are this, or you can start with one peer and they'll discover themselves who their neighbors are, and you give them a node ID. They're integers, they're not strings, and the reason for that is because there's a lot of bit packing of certain data types going on and strings do not do well. And here we can also effectively wait for nodes
19:40
to come onto the system so our cluster is stable and ready to go before we actually do anything else. And by the time we get to this point, our RPC systems are working, nodes are communicating, your clocks have synchronized themselves mostly, and you can actually start adding something called extensions. Now extensions essentially allow you
20:02
to extend your existing cluster. You can do this at runtime, they can be added and they can be unloaded all at runtime without any, with state cleanup and everything else, which makes life a lot easier, especially for testing. They have access to the running node on this local system which allows you to access things like the cluster clock,
20:22
the RPC network as it's called, which is the pre-established RPC connections, and you can essentially make this as simple or as complex as possible, which is essentially what I've done here. So I've created this nice little extension which is absolutely nothing other than print what the current time is, which realistically I could do without,
20:42
but nonetheless I went with it. And this is what the eventual consistency store actually does under the hood, is it's just an extension. And in here we can see that we're passing in a, I can't point that far, but we pass in a memstore, which is our storage trait.
21:01
We pass in our create our eventual consistency extension using this and we pass it to the data cake node and say, okay, go add this extension, give me the result back when you're ready. And in this case, our eventual consistency cluster actually returns us a storage handle, which allows us to do basically all of our lovely key value operations should we wish,
21:23
including delete, put, get, that's about all there is on the key value store. But there are also some bulk operations which allow for much more efficient replication of data. The only problem with this approach is it's not suitable for billion scale databases.
21:42
So if you're trying to make the next Cassandra or Cilla, don't use this particular extension because it keeps the key value, or the keys, sorry, in memory, which it uses to work out what keys have and have not been processed. And the reason for this is effectively because I didn't really trust users
22:00
implementing this on the storage side correctly, which turned out to be a good choice because the amount of unit tests that this failed initially was a lot. And so now we've sort of got this ability to replicate our key values. Our life is a lot easier. In particular, we can actually go as far as essentially saying, okay,
22:21
we've established our data connection, our key values, let's just use Tanteria as our persistence store. And this is effectively the simplest way to do it. And I've made a little demo here, which you can go to that link. I basically abused and slightly ignored certain things, in particular correctness,
22:40
but this will replicate your data. You may end up with duplicate documents because I didn't handle deduping. But in this case, we can fetch, we can delete, and we can index documents with Tanteria, and that's our persistence store. And here you can see we're doing about 20,000 documents in 400 milliseconds in the local cluster.
23:01
Yes. And that is effectively the end. Great, so are there any questions? How long do we have left?
23:21
Oh, how long do we have left? 15 minutes. 15, okay. Nice. Excellent timing. We have a question down here in the front. Oh, yeah. Yes. So I actually kind of saw in there,
23:42
do you have like a way to provide from outside to the Tentivity transaction or Lynx transaction, an external ID that you can use to integrate with a standard storage?
24:00
Change the question maybe in an easier way. Do you have a way to say which level of data has been indexed? Yeah, so in this case, I've sort of glossed over it a little bit because in reality, it's a little bit more complicated when you implement it. So in reality, when you actually implement this, you would probably have a,
24:20
essentially use the replication to replicate the initial documents and then you would have a check mark to essentially work out what documents have and have not been indexed yet. Or you would add some additional step like a write ahead log so that way, you know that as long as the documents are there, you can make sure that your commit point is always updated to the latest thing.
24:44
In LNX, it's actually a little bit different again because the way it creates indexes is they are per checkpoint. So a new index is created every commit effectively. But you don't have to do that and in this method, I didn't. So yeah, it doesn't do it here,
25:02
but you can add a write ahead log and you can basically do anything as long as the trait is implemented. Hello, hello. Hi. Hello. Okay, okay, okay, yeah, so.
25:21
All right, so congratulations for the. I think I can see you. Thank you. Yes, hello. Matching that is happening beyond the search engine.
25:41
Are there any plans to support something beyond the M25 such as vector space modeling or vector search or is this something that you can work on? So let me see if I got that question right. So was that about extending TantrV so if you want to go beyond something like BM25
26:01
or Liebensleid's distance and things like that. Yeah, so just what is the support apart from BM25 that are plans to support in other kind of searches such as distance searches or? Things like, I think things like vector search or word embedding search is still something which is quite far away and would need quite a big push to do
26:22
with TantrV specifically. But if you want to add additional queries or additional functionality, it's quite easy to add with TantrV. So it's actually just a query trait. So one of the things that LNX does, it actually has another, a query made called fast fuzzy which actually uses another algorithm for pre-computing dictionaries
26:41
in order to do the edit distance lookup. And that basically is just involves creating another query. And you can customize effectively all of your query logic, all of your collecting logic and things like that. So providing you're within the scope of the API, TantrV will allow you to implement it yourself.
27:00
Otherwise things like the word embeddings which are a little bit more complicated and require a bit more on the storage side would need to, an issue and a very motivated individual to probably implement that which currently we don't really have.
27:28
So it's pretty little question. On all your sketches, the network, the subject network was fully connected. Is that important? Let me see if I can find which one that was.
27:40
Was it this one or was it this one? Well, on this one, it does not look fully connected, but I'm not sure if this diagram depicts kind of connectivity, connectome or just which messages has actually been dispatched. So I'm going to cross the forbidden white line here
28:02
because we're doing questions. And effectively these are just indicating sending responses and getting things back. So these nodes don't actually in a real system that you could have a network petition here and your node one can no longer talk to node three, it's effectively lost to the ether
28:21
and maybe node two can also not do it. And in this case, it doesn't actually really care. All that you need to do is you need to achieve what's called a consistency level. So which means that if you want to progress, you have to reach that level, otherwise things are counted as not happening. And so in this case,
28:41
if node three is down or can't be contacted, as long as node one can contact node two and node two acknowledges the messages, things can still progress. This is the same with Raft as well. So Raft operates on what's called a quorum, which yeah, but effectively any node, any one node can go down in a three node group and the other two nodes can still progress
29:02
providing they have what's called, what's the majority. So we understand full connection of the network is not an important factor here. Well, it's nice to know. Thank you.
29:22
Thank you for your talk. I see here that there is basically a consistency mechanism for indexing. Do you check as well the data over nodes when there is a search request as well? Say that again, sorry, I didn't quite pick that up. Do you check the data on over nodes when there is a search request, not an indexing request?
29:41
In this case, we have relaxed reads essentially. So we don't do, we're not searching across several nodes and getting the most updated version from that, which is part of the trade off you make with the eventual consistency. You will have that with Raft as well effectively, unless you contact the leader, so you won't have the most update data when searching.
30:03
But one of the things you do have to do if you go with the eventual consistency approach like we do here, is you would need to effectively handle the idea that maybe you will have duplicate documents because something's been resent in the meantime.
30:22
And so you'll need to be able to de-duplicate that when you're searching, or have some other methods of handling it and deleting it from the index. So that means that effectively, every node must have a copy of the data. Like I cannot have five nodes and like a free replica system or something like that. Yeah, so as long as, if you've got like a five node cluster
30:41
and three nodes respond, you can immediately search from, if those three nodes have got the data, they can immediately be searched from effectively if you want. But the other nodes may take a little bit of time to catch up, which is the principle with eventual consistency. They'll eventually align themselves, but they're not all immediately able to reflect the changes.
31:05
Hello. Just a simple one. In hindsight, would you take the rough part? In hindsight, probably not still. And the reason for that is because the current state of the Rust ecosystem with it
31:22
means that there's a lot of black holes effectively around it. And so you either going with an implementation which is very, very stripped down and just the state machine part, or going with an implementation which is very, very trait heavy and is a little bit opaque around what you need to test,
31:41
what you don't need to test and how it behaves under failure. So in this case, it's, I like this approach more because it allowed me to implement things like network simulation, which the RPC framework supports. So we can actually simulate networks failing locally in tests and things like that, which makes me feel a little bit more confident
32:02
than trying to just have the state machine and implement everything and all the handling correctly. But I think in future, yeah, you could use it, but it's just not quite at that state. Great, if that's all the questions, then, oh, come on.
32:25
Should I stand up or? Yeah, for sure. Actually, no. What was I gonna ask? Yeah, so I'm not sure I quite got how, if the engine actually does any data sharding
32:40
or there is a hash ring or? Yeah, so in this approach, for the simplicity of time, really, we're not actually doing any data sharding. Servers are really quite big nowadays, so you can, even for your e-commerce website, you can get a pretty huge server,
33:00
and the biggest issue tends to be replication and the high availability. The data sharding is something that, quick width is something that would be concerned about because you've got so much data, you need to spread it across machines and things like that when you're searching. But in e-commerce, at the point in which you're searching across multiple machines, you're probably going to be looking
33:21
at the higher latencies. So you'd be better off dedicating one machine per search rather than several machines per search, really. So I think that's all of our questions. Can we have another applause?
Recommendations
Series of 2 media