We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Future of Lucene's MMapDirectory: Why use it & what's coming with Java 16 & later?

00:00

Formal Metadata

Title
Future of Lucene's MMapDirectory: Why use it & what's coming with Java 16 & later?
Title of Series
Number of Parts
69
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Since version 3 of Apache Lucene and Solr and from the early beginning of Elasticsearch, the general recommendation was to use MMapDirectory as the implementation for index access on disk. But why is this so important? This talk will first introduce the user about the technical details of memory mapping and why using other techniques slows down index access by a significant amount. Of course we no longer need to talk about 32/64bit Java VMs - everybody uses now 64 bits with Elasticsearch and Solr, but with current Java versions, Lucene still has some 32bit-like limitations on accessing the on-disk index with memory mapping. We will discuss those limitations especially with growing index size up to terabytes, and afterwards, Uwe will give an introduction to the new Java Foreign Memory Access API (JEP 370, JEP 383, JEP 393), that first appeared with Java 14, but still incubating. The new API sounds interesting and will remove all previous issues and limitations, but with Lucene's current design, the first and second JEP incubators (Java 14, 15) would have been hard to implement. In close cooperation between Lucene developers and OpenJDK committers, starting with Java 16, the 3rd incubator will finally be ready to be used from Lucene: A first preview of Lucene's implementation was developed as a draft pull request. Uwe will show how future versions of Lucene will be backed by next generation memory mapping and what needs to be done to make this usable in Solr and Elasticsearch - bringing you memory mapping for indexes with tens or maybe hundreds of Terabytes in the future!
Java appletThetafunktionSoftwareTexture mappingMaxima and minimaCountingDemosceneLocal ringData bufferKernel (computing)MiniDiscCache (computing)SpacetimeResource allocationLuceneDirected setMultiplicationPrice indexComputer-generated imageryOvalContent (media)Process (computing)Computer fileMultiplication signCodeReading (process)Buffer solutionOperator (mathematics)Local ringBitComputer programPosition operatorKernel (computing)System callDirectory serviceType theorySpacetimeCache (computing)Web pageJava appletSubject indexingMiniDiscSemiconductor memoryStructural loadMemory managementFigurate numberArray data structureIntegerVirtual memoryOperating systemImplementationOpen setThread (computing)Program codeDoubling the cubeHeat transferFile systemHard disk driveMereologySpeicherbereinigungPointer (computer programming)Resource allocationDefault (computer science)Standard deviationDuality (mathematics)WordMathematical singularityPasswordWater vaporScripting languagePrincipal idealSoftwareMultilaterationView (database)Bit ratePhysical systemMathematicsYouTubeCASE <Informatik>Auditory maskingSampling (statistics)DeterminantSound effectInclusion mapLattice (order)Direction (geometry)PhysicalismComputer configurationCrash (computing)Latent heatKey (cryptography)XMLUMLComputer animation
Java appletImplementationDirected setKernel (computing)MiniDiscPhysical systemWeb pageCache (computing)Depth-first searchSystem callBlock (periodic table)Operating systemWindowFile systemDirectory serviceJava appletSubject indexingPhysical systemTotal S.A.SynchronizationPosition operatorRandom accessSoftwareBit rateCache (computing)Process (computing)Multiplication signMultiplicationTransformation (genetics)Similarity (geometry)Right angleCrash (computing)Disk read-and-write headLaptopChainPermanentCircleSemiconductor memoryPrincipal ideal domainDefault (computer science)Kernel (computing)32-bitMemory managementOperator (mathematics)Term (mathematics)Query languageBuffer solutionHeat transferComputer fileQuicksortImplementationDifferent (Kate Ryan album)Computer animation
CodeFile systemSemiconductor memoryCache (computing)Operating systemImage resolutionPhysical systemCodeVirtualizationKernel (computing)Java appletSpeicheradresseComputer fileProcess (computing)Computer animation
Virtual realitySemiconductor memorySpeicheradresseSpacetimeProcess (computing)Web pageJava appletLogicComputer programPrice indexRead-only memoryLevel (video gaming)LuceneSubject indexing10 (number)AbstractionoutputFunction (mathematics)Computer fileSequencePointer (computer programming)Tap (transformer)File systemSemiconductor memoryFunction (mathematics)SeitentabelleAbstractionSubject indexingSpeicheradresseWeb pageSpacetimeOperating systemPhysical systemProcess (computing)TrailData storage deviceDressing (medical)BitLogic programmingTranslation (relic)Engineering physicsPerturbation theoryGraph (mathematics)InformationAddress spacePointer (computer programming)Ocean currentPhysicistVirtualizationSuite (music)Table (information)Cache (computing)Data managementIdentity managementMiniDiscLogic gateQuadrilateralShared memorySoftwareMedical imagingWebcamComputer hardwareMusical ensembleConstructor (object-oriented programming)Web 2.0Electronic visual displayArithmetic meanPhysicalismVirtual memoryIntelligent NetworkBefehlsprozessorTouch typingMaxima and minimaServer (computing)Game controllerMikroarchitekturMappingComputer fileAreaJava appletInformation securityKernel (computing)CASE <Informatik>UsabilityMultiplication signTerm (mathematics)Operator (mathematics)Field (computer science)Query languageDirectory serviceResultantDirection (geometry)outputBuffer solutionCuboidDiagramComputer animation
Data bufferTexture mappingShift operatorBuffer solutionLimit (category theory)Exception handlingComputer fileQuery languageCharacteristic polynomialSynchronizationThread (computing)AreaSemiconductor memoryPointer (computer programming)Java appletPhysical systemModule (mathematics)Integrated development environmentError messageRun time (program lifecycle phase)Core dumpRevision controlServer (computing)Asynchronous Transfer ModeMixed realityFactory (trading post)Computing platformDefault (computer science)Open setPrice indexLattice (order)Virtual memoryMusical ensembleMaxima and minimaWeb 2.0Logistic distributionSatelliteWater vaporComputer fileDefault (computer science)Factory (trading post)Multiplication signSubject indexingSoftwareMetreBitMehrfachzugriffSpeicheradresseSeitentabelleRandom accessIntegerMereologyThomas BayesSinc functionJava appletCodeDoubling the cubeEqualiser (mathematics)PlanningAxiom of choiceFreewareCASE <Informatik>TwitterMappingNewsletterEscape characterInternetworkingInheritance (object-oriented programming)Group actionOpen sourceGame controllerEvent horizonCubic graphLocal ringMoving averageElasticity (physics)WindowEntire functionWhiteboardLevel (video gaming)Exception handlingDisk read-and-write headPointer (computer programming)Inclusion mapAreaSemiconductor memoryInformationHessian matrixAssociative propertySocial classUniformer RaumBuffer solutionPower (physics)Process (computing)Query languageBoundary value problemCharacteristic polynomialInterior (topology)LogicMemory managementPhysical systemSynchronizationOpen setWritingRight angleAddress spaceSpacetimeSpeicherbereinigungField (computer science)Shift operatorDivision (mathematics)Thread (computing)Auditory maskingComputer animation
Price indexPrincipal ideal domainVirtualizationSemiconductor memoryHydraulic jumpCuboidSpeicheradresseDressing (medical)Subject indexingVirtual machinePrisoner's dilemmaAddress spacePhysical systemFunction (mathematics)Elasticity (physics)Information overloadMemory managementVirtual memory
Subject indexingCodeJava Virtual MachineJava appletInterface (computing)Read-only memorySpacetimeSpeicheradresseSemiconductor memoryData bufferOverhead (computing)Thread (computing)Instance (computer science)Color confinementOpen setQuery languageHeat transferCloningState of matterBuffer solutionCompilerInformation securityMaxima and minimaOperations researchSpeichermodellPrice indexDefault (computer science)Reduction of orderSoftware testingLuceneException handlingIntegerPosition operatorFluid staticsBoolean algebraGraph (mathematics)ImplementationUsabilityKernel (computing)Computer hardwareControl flowCodePoint (geometry)CASE <Informatik>Computer fileFlow separationQuery languageProjective planeMultiplication signShared memoryBitGoodness of fitState of matterDirectory serviceVirtual memoryColor confinementThread (computing)View (database)Closed setAddress spaceVirtualizationSpeicheradressePhysical systemSoftware developerArray data structureJava appletSemiconductor memorySet (mathematics)Interface (computing)Process (computing)Moment (mathematics)Buffer solutionSoftware testingImplementationException handlingSubject indexingWindowAsynchronous Transfer ModePointer (computer programming)DemosceneMereologyRevision controlComputer programContrast (vision)Connectivity (graph theory)Virtual machineUltraviolet photoelectron spectroscopyFacebookLimit (category theory)Data storage deviceNeuroinformatikMobile appWebsiteSoftware bugBeta functionInternetworkingPosition operatorMusical ensembleHome pageSource codeComa BerenicesEmailWave packetCentralizer and normalizerSummierbarkeitCategory of beingWordWeb 2.0Code division multiple accessUniverse (mathematics)Level (video gaming)Dressing (medical)Length of stayFreewareField (computer science)Compact spaceTime zoneExpandierender GraphDisk read-and-write headStatement (computer science)Electronic data processingWeb pageDirection (geometry)Group actionLine (geometry)Stress (mechanics)Computer animation
View (database)Point (geometry)Elasticity (physics)TrailVector spaceData dictionaryFlagMemory managementElectronic mailing listFeedbackTerm (mathematics)MathematicsComputer fileBuffer solutionSubject indexingCombinational logicProcess (computing)Line (geometry)AlgorithmAreaVirtual memoryBitTraffic reportingProjective planeSemiconductor memoryWeb pageForm (programming)CASE <Informatik>
Group actionBitContent (media)
XMLUML
Transcript: English(auto-generated)