We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Search in the Cloud: separation of compute and storage

Formal Metadata

Title
Search in the Cloud: separation of compute and storage
Title of Series
Number of Parts
64
Author
Contributors
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
The traditional architecture of a search cluster, with nodes providing compute and persistent storage, ends up being more expensive to run than it should, especially in the context of public cloud and when dealing with unpredictable traffic patterns/load. This presentation is about modifying the implementation of SolrCloud so a single copy of each index is stored on shared storage (for example S3) and accessed by all nodes for reads and updates. The nodes are then stateless, which means any one (or all) of them can be turned off at any time without data loss. New nodes added to the cluster get their data from the shared storage and not from other nodes. Note that many aspects of this new architecture apply to other search clusters, and no prior knowledge of SolrCloud is assumed. Such an approach is a good match for running Search on a public cloud infrastructure, as it allows for compute and storage to be scaled independently and decreases the overhead of bringing a new node up in the cluster. The challenges faced in the process of separating compute and storage will be detailed. The hard problems concern updates (indexing), and protecting the shared storage from conflicting concurrent updates, preventing data loss or overwrites yet allowing concurrent updates. Tradeoffs for integrating such a large conceptual change into an existing system will also be discussed. The existing code has limitations that will be presented, as well as a high level design and plans for lifting them. The current code is available (see SIP-20). The aim is to eventually merge these changes into the main Solr branch.