We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Building a distributed search engine with tantivy

Formal Metadata

Title
Building a distributed search engine with tantivy
Subtitle
How lnx is solving the challenges of builing a distributed search engine in Rust
Title of Series
Number of Parts
542
Author
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Follow me as I walk through my journey of building lnx, a distributed search engine written in Rust akin to Elasticsearch or Algolia that aims to be faster and more efficient using tantivy. I cover the challenges and solutions I encountered while developing lnx over the last year, the tradeoffs made and how you can build your own search engine using the tools the Rust ecosystem provides. In the first part of the talk we'll look at what a search engine like Algolia or Elasticsearch does, how tantivy works, what things we need to do in order to make it suitable for user-facing search and how we implement it into our code to build our own application. In the second part of the talk I talk about the biggest issue you run into when building a search engine or any other system designed to store data... Fault tolerance and replication, together we'll go through the challenges of implementing this for a search engine using tantivy, keeping the performance competitive, reliable and the tradeoffs that are made to make it work. We'll look at some existing implementations like Raft and take a dive into the world of eventual consistency! With any luck, you'll have a better understanding of how we can build services like search engine using libraries like tantivy along with solving technical challenges like consensus and replication for distributed systems in Rust.