#bbuzz: Cross DC replication in Apache Solr - Beyond just forwarding data
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 48 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68782 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Berlin Buzzwords 202046 / 48
14
19
26
43
47
00:00
SolrReplication (computing)GEDCOMDifferent (Kate Ryan album)Replication (computing)Computing platformMultiplicationClient (computing)CASE <Informatik>Physical systemQuicksortCuboidGene clusterService (economics)ConsistencyScalabilityDependent and independent variablesMultiplication signSound effectArithmetic meanInternet service providerData centerAdditionError messageNatural numberConfiguration spaceComputer hardwareMathematicsComputer architectureCohesion (computer science)Point cloudStandard deviationDiagramSoftwareLoop (music)Process (computing)Form (programming)Revision controlConcurrency (computer science)Centralizer and normalizerCartesian coordinate systemSinc functionAsynchronous Transfer ModePosition operatorExtension (kinesiology)Local ringPlanningTheoryMarginal distributionTable (information)Server (computing)Point (geometry)Data recoveryTerm (mathematics)Special unitary groupComputer animationXMLUMLDiagramProgram flowchart
10:14
CodeTerm (mathematics)State of matterMathematicsSystem administratorSubject indexingSelectivity (electronic)Musical ensembleData centerMereologyPhase transitionPhysical systemCuboidReplication (computing)Mechanism designQuery languageComputer animation
12:22
CodeTerm (mathematics)CoprocessorQuicksortChainData centerDatabase transactionThread (computing)Gene cluster
13:14
ConsistencyBand matrixConfiguration spaceLoginDifferent (Kate Ryan album)Utility softwareDatabase transactionThread (computing)DiagramData centerUltraviolet photoelectron spectroscopySet (mathematics)Queue (abstract data type)Consistency
14:30
MereologyPhysical systemDatabase transactionData centerLimit (category theory)RippingQueue (abstract data type)
15:22
Cache (computing)Computer configurationLoginConfiguration spaceFreewareData centerLevel (video gaming)Database transactionMetric systemComputer fileRouter (computing)Thread (computing)CASE <Informatik>Multiplication signCuboidPhysical system2 (number)
17:04
Computer configurationMultiplication signProcess (computing)Table (information)MereologyRevision controlInstance (computer science)Dependent and independent variablesReplication (computing)Data center
18:21
Data centerException handlingError messagePhysical systemSynchronizationConsistency
19:12
Replication (computing)ConsistencyDrop (liquid)Source codeStructural loadQueue (abstract data type)NumberCASE <Informatik>Concurrency (computer science)CountingQuicksortCuboidGreen's functionBlock (periodic table)Point (geometry)Data centerDataflowError messageRepetitionConsistencyInformation9 (number)AlgorithmConfiguration spaceForm (programming)Multiplication signReplication (computing)Mobile appRevision controlField (computer science)CoprocessorSubject indexingClient (computing)Process (computing)Insertion lossPlug-in (computing)DiagramComputer architectureTimestampPhysical systemCartesian coordinate systemPotenz <Mathematik>Existence
25:10
Source codeMultiplicationSurjective functionData centerQuicksortGene clusterSubject indexingConfiguration space
25:57
Revision controlIntrusion detection systemQuery languageQueueing theory
26:27
Revision controlDemosceneCASE <Informatik>Term (mathematics)ConsistencyProcess (computing)Error messageData centerAddress spaceStrategy gameBasis <Mathematik>Regular graph
27:39
Revision controlConsistencyGene clusterMechanism designMereologyConsistencyPhysical systemCASE <Informatik>
28:35
ConsistencyDataflowNumberData centerConsistencyBasis <Mathematik>Physical systemRevision controlMechanism designTable (information)State of matter
29:37
Revision controlConsistencyNetwork topologyRevision controlPhysical systemArray data structureNetwork topologyData centerNumberImplementationMultiplication signProgram flowchart
30:18
ConsistencyNumberRevision controlRow (database)Right angleLimit (category theory)Physical systemMultiplication signMereologyDifferent (Kate Ryan album)Data centerQueue (abstract data type)FlagCASE <Informatik>ConcentricImplementationProgram flowchart
32:11
Condition numberDependent and independent variablesMechanism designService (economics)Queue (abstract data type)JSON
32:46
Parity (mathematics)AdditionField (computer science)Self-organizationService (economics)Disk read-and-write headQueue (abstract data type)Data centerConfiguration spaceExistenceSpacetimeMiniDiscConsistencyPolar coordinate systemJSON
33:32
Condition numberDatabase transactionMultiplication signPhysical systemMetric systemQueue (abstract data type)JSON
34:08
Near-ringPlanningBand matrixReplication (computing)
34:39
Form (programming)CASE <Informatik>Level (video gaming)CodeSoftware testingComputer animation
35:15
Virtual memoryRevision controlIntegrated development environmentCASE <Informatik>Computer animationMeeting/Interview
36:40
XMLUML
Transcript: English(auto-generated)
00:11
I think I was here. So hi, everyone. Welcome to my talk about cross-TC replication in Apache Solar, beyond just forwarding data.
00:20
And thank you, Varun, for the introduction. So the agenda for the day, and thank you for bearing with us, the agenda for the day is I'm going to start off with talking about the basic architecture of Solar Cloud, then move on to talking about the need for disaster recovery, and then I'm going to talk about different approaches to achieving cross-TC in Solar,
00:43
and I'm going to spend most of the time for this talk on that one specific thing. And then wrap it up by talking about what I believe could be the future of how to achieve cross-TC in Solar in a manner that's cohesive and works for most people.
01:02
So a very simplified version of a solar architecture diagram is something that looks like this. You've got shards that have leaders, followers, and they talk to each other, and they pretty much play the same role, other than the fact that every incoming update is routed through the leader
01:21
who's also responsible for versioning and doing a bunch of other managerial things as far as handling documents and data is concerned. So like any distributed system, it seems fairly easy to run and manage. And the shard replicas, they provide scalability
01:43
and availability, so if one of them goes away, the other one comes in, assumes leadership. Also, if you need to support more traffic, you can always add more replicas there. And so that kind of solves most of the problem for standard use cases.
02:01
But when it comes to critical systems, it's kind of a no-go to just rely on those basic things that a basic solar cloud setup provides, because any critical system would need to account for things like hardware failures, natural calamities,
02:21
and most importantly, and the most common of the reasons as to why things go down, human errors. And there are multiple solutions to solve these problems, the most common one being backup and restore. Well, it doesn't really do completely off the job. Also, when you're backing up and restoring, you should be backing it up to a data center
02:43
that's outside of your primary data center, only because if something were to happen to your data center, you need somewhere to restore your data from. So that comes with a lot of caveats in addition to the fact that it doesn't provide everything that you might need from a solution
03:03
that supports a critical system. So there comes cross-DC replication, which not only provides guarantees around availability, but also provides things like scalability and reduced latency. So for example, if you were running a service
03:21
that was supposed to only cater to North America and you had your data sitting in data centers across the US, all of a sudden, you had to provide some sort of support to order some, you were launching the same service, making this data accessible to people in Europe. You could set up a cross-DC replicated sort of cluster
03:42
in a data center in Europe, and that would allow for you to have lower latencies for users who are in Europe. So what is cross-DC replication? In the most simplest of terms,
04:02
it's nothing but a means to achieve the sort of mirroring effect across data centers for solar clusters. So anything that you ingest in data center one should show up in data center two for it to be searched on, retrieved or whatever. So that's the most, that's a basic understanding,
04:21
the basic need of why you might want CDC cross-data center replication. So now we're gonna talk about the different approaches to achieving cross-data center replication.
04:41
The first one, the client-based replication is something that we started off with an Apple, and it kind of predates all other solutions because it doesn't really require too much to be built on. It does rely on your users being wise. It does rely on your users taking responsibility.
05:03
So this solution kind of predates everything else that was built to work out of the box or things that were built to support something like this. If you look at what happens in a client-based data center replication,
05:22
as obvious, it's the responsibility for everything lies on the client in such a case. And the client is basically responsible for managing external versions. Solar does provide some form of, not some form of, but a pretty good optimistic concurrency.
05:41
And so as long as you're versioning your documents correctly, Solar's gonna take care of things for you. But the versioning has to come from the client in a setup that relies on the client to provide cross-data center replication. And not only that, in this case,
06:01
client also ends up managing request failures and retries. So the challenges in such a system, though, are if you had a client sending data and say it succeeds in the local data center,
06:22
so if you had a client sending data to your data center and it succeeds on one of the data centers, but does not succeed on the other data center,
06:40
it's the client's responsibility to make sure that that failure is taken care of and the data is synchronized. And consistency is checked and users are alerted when the data doesn't make it through to across all the data centers. Not only that, all the process has to be synchronous
07:02
to some extent. The reason being if the client sends data to data center one, hears back, gets positive back, sends data to data center two, doesn't hear back, it's its responsibility to close that loop and either delete that data from data center one or wait until data center two also has that data
07:22
to positively acknowledge that request. So it kind of makes it rather difficult for the client to operate in such a mode. Also when a data center goes down, so for example, if data center two were to go away due to network outage, weather issues,
07:41
whatever it might be, the client is going to get stuck and that might require changes in the configuration for the client to make sure that you can go ahead and forget those problems and ignore those problems for the client to understand that it's okay
08:03
to ignore those failures might require a config change in such a case. So while the system looks very simple to begin with, in my opinion, the cost to operate this is pretty high, especially if you are going to end up using it at any given point in time.
08:22
And that comes from our experience having asked our users to start off with this because of lack of any other solution a long, long time ago. And it's kind of still okay if you're doing this by yourself for a sort of cluster that you are the user of,
08:41
but it kind of gets really difficult when you're just a provider of a platform that others use. Because as I said in one of my talks, I think last year, sorry, I've said in one of my talks last year,
09:06
that when you're finding a platform, your users are not your best friends. So they want a great stable up and running system. They want the best performance to come out of it, but they're not, they're willing to take you down
09:22
if that means a marginal improvement for their use case. So they're not bothered, but they will bother you when it comes to resolving a problem. And so in such a system where you're relying on your client to send in this data and make sure that it's consistently sent
09:42
across multiple data centers and cleaned up when there's a need to do so, that problem gets amplified when you're running a platform. So I really feel that in platforms, this value kind of goes not to zero, but at times to negative,
10:01
because you're firefighting and trying to figure out how to now heal and fix these problems that were created by users who did not understand the implication of what they were doing. So that brings me to the next approach. So while that was something that I know
10:22
we at Apple were doing, and a few other companies were also doing things that way, a bunch of people in the community realized there was a need for a genuine cross data center replication solution within Solr. And so a few folks in the community,
10:43
Eric Erickson, I guess, primarily went out and started working on a cross-DC replication mechanism that would be supported by Solr out of the box. And the way it works is any request that comes in, and remember, most of these requests that I'm looking at
11:02
are either admin requests that make changes to the state of the system, or they're update requests. Anything outside of that does not really need replication. For example, select queries have no need to be replicated
11:21
to go anywhere other than the data center that they're sent to. And so do collection admin requests that cause no state changes to the cluster itself. So when a request comes in, the data centers are working in isolation, but they do have some knowledge about each other.
11:43
So in this example, the shard one leader is going to not only index things locally and send them out to the follower, but is also going to somehow manage to send this data across the pipe so that a cross-DC replication happens
12:02
to the leader of the corresponding shard in another data center. Now, if there are more than one of these kinds, this leader is going to ensure that it reaches every other leader that should have this data, this document.
12:23
So in terms of the code path, the way it really works is the request comes in, and for all of you who are aware of how Solr works, there's something called the update request processor chain, which is nothing but a pipeline of sort that works to process every incoming update.
12:42
And so on the leader, it comes in, gets processed through the update request processor chain, and then gets dumped into the transaction log. Transaction log in Solr is basically, it's a log of all the updates that are coming in.
13:02
And there is a thread that's running outside of the leader called the CDCR replicator thread that tracks what are the replicated data centers, data center clusters, or CDCR clusters, and feeds data off this transaction log, and based on some basic configuration
13:22
around bandwidth utilization and other things, reads data from the transaction log and pushes it out to the leader of the target DC, the leader on the target DC. The difference on the target DC being, it's smart enough to make sure that it doesn't,
13:40
this data does not come back to the primary DC that originally sent this data. So in that diagram, what we see is that the transaction log basically is acting like a queue because everything that's indexed locally is going into the transaction log,
14:01
kind of getting buffered there, and then being read by the thread that's responsible to send out all of these updates to everyone who's supposed to be replicating. And we do get an eventually consistent DR cluster that supports both active-passive and active-active setups but with a bunch of caveats,
14:21
especially because the versioning in Solr CDCR solution relies on clocks. And so if you have data centers who have clocks out of sync, that might be kind of complicated or scary. The best part in my opinion here is that it's standalone,
14:41
which basically means it translates to no external dependency. You don't really need anything else running on the site. You don't need to rely on a third-party system. Also, because it's part of Solr, you do get community support from others who are using the exact same solution instead of having their own custom solutions,
15:01
trying to replicate data from one place to the other. So, but obviously this comes with the limitations. One of them being the transaction log, considering it's used as a queue, if your target data center for some reason goes away,
15:23
your primary data center is going to accumulate all the incoming updates in its transaction log without purging them. Only because at CON, that's the only place from where to replay and replicate stuff to target data centers. And that certainly might run the autotisk
15:42
on your primary data center. The configuration on this one is not easy for some reason. There's collection level configs and cluster level configs and a bunch of other configs that kind of make it harder to use in my opinion. And then the transaction log approach doesn't really catch up because you're looking at a file-based transaction log
16:02
that's sitting on the system, which is okay for small use cases that don't have a ton of data coming in every second. But if the target cluster is gone and you have a ton of data coming in, the transaction log will grow to a size that the CDCR thread is not going to be able to catch up with.
16:23
So it's kind of a no-go for any high throughput situation. And then the support for implicit routing is missing, which what that basically means in Solr is if you are not using Solr's composite ID routers or hash-based routers,
16:44
if you're trying to send data explicitly to a specific shard, that will not get replicated by the CDCR that comes out of the box with Solr because that's not supported. The metrics that CDCR supports also get reset on restart.
17:03
A lot of those metrics are okay to be reset, but there are times when you might need historical data, which will go away as soon as your Solr instance is bounced. And obviously there's an extra burden on chart leaders because they're responsible for replicating all of this data across to the other data center.
17:23
So when I spoke about all of that, one interesting thing to know is that we never really used a table CDCR that was offered by Solr, not because we had concerns. I mean, there were certainly things that we were concerned about, but because while we were on the first approach
17:44
to solving CDCR by using client-side replication, we had already done a lot of work and we already started working on our own internal solution that was closely coupled with what we needed. And while that happened,
18:01
the community-based CDCR also was introduced, but we already had our own version of CDCR that we thought was doing the job pretty well and was also addressing a lot of these problems that I just spoke about. So this third and the last part is pretty much that covering and touching upon those things.
18:21
So we've been using this for, I think, about four or five years now. And what it provides is it provides an easy way to replicate in ways across end data centers in an active-active manner. It also allows for, provides retries and error handling
18:45
for failed requests across data centers. And consistency isn't, it wasn't, sorry, that's wrong. I mean, consistency in this system is checked.
19:01
It's not guaranteed, obviously, because if something happens and you get out of sync, the least that would happen is that you would get to know about consistency. We have systems in place that would allow us to fix it, but it, at the very least, informs us of consistencies when they happen.
19:22
And insights were missing from the previous cross-DC replication, things like latency, consistency, errors and retry counts. But in this case, we built a system where we get all of those numbers as well. So how does this setup look?
19:43
So this is the basic architecture diagram, basic architecture diagram for what we use. And we use Kafka, we built a cross-DC replication using proprietary queues, but also Kafka.
20:01
And when a request from the client comes in, it comes into the primary or whatever is this targeted sort of DC that it comes into. That is its own local zookeeper and it has its own local queue. So if you look at the closely coupled orange, yellow,
20:20
and the green boxes, they're together in one data center. And then there's a mirror in the middle that mirrors this onto another queue and sends this data to be consumed by something called the cross-DC consumer, which is a standalone app. And we're gonna get to that in a bit. And that runs in isolation again,
20:42
and has no idea about the existence of the data center that this request originally came into. So what this gives us is the ability to isolate the queue and the mirror from all of the stuff that is solar.
21:05
So I see we're running short on time now. So data flow in this cross-DC plugin kind of is implemented as an updated request processor. When a client sends in a request, it comes in with a doc version and an ID.
21:21
And the receiving data center accepts it only if the received doc version is more than the already existing doc version for that document, if that document exists in the index already. And then solar obviously assigns its own version value, which is used internally by solar.
21:41
But what this updated request processor does is it strips off this version field and inserts this into the queue, into the source queue on the source or the originating data center, which is then replicated and copied over to the other data center by the mirror in the middle.
22:07
So it's then received by the cross-DC consumer and the job of the cross-DC consumer is pretty straightforward. It's a very straightforward, simple app. What it does is it reads from the destination queue,
22:21
which is isolated from the source queue because the mirror just mirrors stuff based on config. And then it reads that data and it tries to send these updates to solar. Now, those requests could in isolation succeed or fail.
22:40
We over time learned by making a lot of errors as to what makes sense to be retried and what makes sense to be resubmitted or discarded. So one of these things happen when a request is processed by the cross-DC consumer. They either are successful requests
23:02
or they're failed requests that should not be retried anymore, or they're temporary failures that could be fixed. And so these are then resubmitted onto a separate topic. And we submit these with a timestamp on when this was lost retried, how many times it has been retried so far,
23:21
so that when it comes in again to be retried, we know how many times has it been retried to limit that or alert if a request has been retried too many times without any success. And we put some form of an exponential back off algorithm
23:42
in there to make sure that the same request doesn't get retried too often. And the two kinds of requests that primarily we finally ended up rejecting outrightly are the 409s, which is basically something, is a case where solar would go out and say that based on optimistic currency, the version that you're sending for this document
24:01
is older than what I already have, which means there's no need to send me this update. I already have a newer version of this document. Or in case of a collection creation command that fails because the collection already exists. Because if the way it's set up right now is the primary DC submits these requests into the queue,
24:23
if for some reason it failed or passed and two of these requests were submitted to the primary DC and made it into the queue, they would be received by the receiving DCs. They should just be rejected because they're kind of item potent. They wouldn't cause any damage, but there's no point retrying them.
24:47
And the other good thing here is the n-way replication, which kind of makes it easy and isolates everything. So if you look at the solar cluster at the top,
25:01
the yellow, red and green boxes that are around each other are kind of, they work in isolation. So the solar cluster writes to a local source topic. Solar cluster also works along with across DC consumer, which reads from a destination topic. And then there's a mirror sitting in the middle
25:21
that's configured to just read off of a source topic and copy this data onto multiple destination topics. So if you wanted to add another data center, yes, there's some effort that would be required to bootstrap these, but once you have your indexes bootstrapped,
25:42
all you need to do is set up or add a config to the mirror, set up a destination topic, start across DC consumer, and you'd have a solar cluster that would now be receiving updates from other solar clusters. And also mirror is something that's lightweight and is kind of provided by most queue systems.
26:07
Handling of delete requests is a little different because they're not versioned, delete by queries are not handled, they're not supported. Delete by IDs are however supported, but it works as a tombstone.
26:23
And this tombstoning is very different from Lucene tombstones in a sense that the document is left there as per, if you were to compare it to a Lucene document, the document is left there active and alive in terms of Lucene,
26:41
but in terms of the business use case, it's marked as deleted so that none of the requests come back with these documents. The reason why this was done this way was to ensure that any accidental delete request that was sent to a primary DC does not blindly delete stuff and replicate that stuff
27:01
because that would defeat the purpose of having a DR strategy because it wouldn't address that human error. So you could go back and undelete these, but we generally never need this, which allows us to run back jobs to clean up all of these tombstone documents on a regular basis.
27:26
Inconsistency detection, which is kind of super critical because if you have a cross DC set up, up and running, but you don't know if the data across your data centers is actually replicated well or not,
27:40
you don't really know whether you can use it. And in case of complicated systems that might lead to a situation where you're investing a ton of money into large DR clusters that are sitting with data that is inconsistent and unusable. So you didn't need an inconsistency detection mechanism
28:01
at the very least to make sure that all the money that you've invested in setting up a DR cluster is actually ready to be used or ready to... It's kind of making sense. So as I said, detection is the more essential part of it.
28:22
Healing, not so much because there are multiple ways to heal. You can replicate again. Also, if you're having multiple, having this problem over and over again, that's a bigger problem to look at as to why you're dropping data or why do you even have this inconsistency? But otherwise, all you need
28:41
is a detection mechanism at the very least. And this detection has to happen not on just merely the basis of number of documents, which a lot of people do as a very basic rudimentary approach to figuring out if the data across two different DCs is the same or not. But what we do is that we check on every document,
29:01
a document ID and version table to make sure that that peer exists across all DCs. And the interesting thing in designing a system like this is to account for all the data that is still in flow. So when you look at the state of a primary data center
29:23
or a data center X, when you look at another data center that's supposed to be a mirror of this one, there would be data that would have gone from and to between or between these two data centers causing for a different version of your data snapshot
29:41
that you'll be looking at. So when you're designing such a system, you need to be aware of this. And I think we certainly don't have enough time to look at this, but the way we solve this problem is by using Merkle tree based implementation. And the way it works is every document,
30:04
so we have a bunch of seeds. We define how many seeds do we have and we define the number of buckets. Seeds are just randomized. And when we define seeds and buckets, this is basically a two-dimensional array. Every document in a data center is then hashed
30:22
using each of those seeds and then marked in each of those buckets. So you see one blue cross in each of those rows there that represents say doc one or the bucket that doc one belonged to when they were hashed using seed one, seed two and seed three. So you get this array based implementation
30:44
based on your document, your seed and your number of buckets. And you do the same exercise in your secondary data center and you compare these two to figure out the difference or the missing data between both of these. Now, the way we go back to look at
31:01
whether they're consistent or not is not to just rerun this after say two minutes. But what we do is we have a predefined time limit and say, okay, we know the documents that were not visible in the last run. So what we wanted to concentrate on now is did those documents ever show up?
31:22
If they did show up, are they there with the version that we saw them originally in the primary data center with or with a newer version? Both of these cases are good cases. But if they either did not show up or showed up but had a version that was lower than what we saw at the primary data center,
31:41
that's a red flag. That's when we kind of alert for like, hey, there seems to be an inconsistency. It's something needs to be done about it. And the good part about that approach also is of using such a system is you have a key-based system. So you can always replay things
32:02
if you think you missed out on something or you dropped data. It's always possible to go back and replay if all of this data is sitting on a system that you've prepared to work with your solar cluster rather than completely relying on just your solar cluster.
32:20
So to summarize that approach or this approach, there's segregation of responsibility, something that we fairly wanted to do so that solar is not left with taking care of stuff outside of search. And queuing and mirroring are managed by third party queues. So what that means is you don't have to really even run it yourself.
32:41
You could be using and consuming a service that is either provided by someone else. It could be a publicly provided queuing mechanism or it could be another team that provides the service to you in your organization. And there's retrying of field requests in addition to the possibility
33:01
of just retrying everything but rewinding the head on your queue. And adding another DC is easy because all of these data centers are kind of agnostic of each other. The only person who really knows about the existence of each of those DCs is either the consistency checker
33:21
which runs in isolation or the mirror config. And the Crust DC consumer, as I said, doesn't use solar resources. So you're not going to run out of the disk space because of transaction log exploding or the leader being bombarded with a ton of updates will not translate into, okay,
33:41
now I'm kind of contending for resources where I want to send this data across using CDCR but at the same time also process them locally and make sure that this data is up and running stable on the primary DC to begin with. And then you could have poor visibility into telemetry metrics if you use a queue
34:02
that is managed by a third party system. So to just wrap it up, the future, we tend to have a discussion in the community and talk about converging the solution so that we have fun single end-to-end solution.
34:23
We didn't have the bandwidth to do that so far but the plan is to do that in the near future if that's something that other folks in the community are interested in. So watch it for the JIRA's and if you're interested in cross-DC replication in solar,
34:41
please reach out and participate in the community. And yeah, any form of participation, it doesn't have to be just code level participation but design, sharing your use cases, challenges that you face is as valuable as anything else
35:01
as the code or providing tests. So yeah, participate. That's about it. Thank you so much. Hey Anshum, thanks for the talk. There are like a few questions on channel but essentially we're gonna take one question out of those and then have the rest,
35:22
have you answered the rest in the GTC breakout room after the talk? So I'm gonna help you answer it in a short version because it seems like the questions revolve around the JIRA pointing to remove CDCR from solar. So if you have like a short answer to why there's a JIRA like that
35:42
and like people have questions around that, we can answer that now and then you can probably- I can try and answer that in as few words as possible for now and then we can elaborate in an offline discussion. And yeah, because of the challenges that I mentioned
36:02
that we realized that the current CDCR has, it kind of seems like it needs to be either removed or redone, in either case, removed from the way it currently stands to have something that is usable in a practical and set up in a practical environment.
36:22
So yeah, that's a reason why I said the feature of this is for the community to discuss and come up with a solution that converges and actually solves the problem rather than have a feature that exists but comes with so many drawbacks that it makes it impossible to use in the real world.