We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Search in the Cloud: separation of compute and storage

Formale Metadaten

Titel
Search in the Cloud: separation of compute and storage
Serientitel
Anzahl der Teile
64
Autor
Mitwirkende
Lizenz
CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
Identifikatoren
Herausgeber
Erscheinungsjahr
Sprache

Inhaltliche Metadaten

Fachgebiet
Genre
Abstract
The traditional architecture of a search cluster, with nodes providing compute and persistent storage, ends up being more expensive to run than it should, especially in the context of public cloud and when dealing with unpredictable traffic patterns/load. This presentation is about modifying the implementation of SolrCloud so a single copy of each index is stored on shared storage (for example S3) and accessed by all nodes for reads and updates. The nodes are then stateless, which means any one (or all) of them can be turned off at any time without data loss. New nodes added to the cluster get their data from the shared storage and not from other nodes. Note that many aspects of this new architecture apply to other search clusters, and no prior knowledge of SolrCloud is assumed. Such an approach is a good match for running Search on a public cloud infrastructure, as it allows for compute and storage to be scaled independently and decreases the overhead of bringing a new node up in the cluster. The challenges faced in the process of separating compute and storage will be detailed. The hard problems concern updates (indexing), and protecting the shared storage from conflicting concurrent updates, preventing data loss or overwrites yet allowing concurrent updates. Tradeoffs for integrating such a large conceptual change into an existing system will also be discussed. The existing code has limitations that will be presented, as well as a high level design and plans for lifting them. The current code is available (see SIP-20). The aim is to eventually merge these changes into the main Solr branch.