Uncoupling Systems
This is a modal window.
Das Video konnte nicht geladen werden, da entweder ein Server- oder Netzwerkfehler auftrat oder das Format nicht unterstützt wird.
Formale Metadaten
| Titel |
| |
| Serientitel | ||
| Anzahl der Teile | 66 | |
| Autor | ||
| Mitwirkende | ||
| Lizenz | CC-Namensnennung 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. | |
| Identifikatoren | 10.5446/46631 (DOI) | |
| Herausgeber | ||
| Erscheinungsjahr | ||
| Sprache |
Inhaltliche Metadaten
| Fachgebiet | ||
| Genre | ||
| Abstract |
|
Ruby Conference 20185 / 66
5
10
13
14
17
18
21
22
26
29
37
45
46
48
50
51
53
54
55
59
60
61
63
65
00:00
SystemprogrammierungVideokonferenzFormation <Mathematik>CodeKartesische KoordinatenSelbst organisierendes SystemComputersicherheitBitTurm <Mathematik>Exogene VariableRechenschieberSystemprogrammierungProzess <Informatik>JSONXMLUML
01:53
Shape <Informatik>ParserProzess <Informatik>Kartesische KoordinatenOnline-KatalogGraphfärbungDatensatzSystemprogrammierungKonfigurationsraumKlasse <Mathematik>ProgrammbibliothekExogene VariableValiditätClientCodeMailing-ListeInternetworkingHilfesystemXML
04:06
SystemplattformPunktwolkeHilfesystemProdukt <Mathematik>RechnernetzMengentheoretische TopologieBandmatrixUnendlichkeitPasswortWiderspruchsfreiheitSystemprogrammierungInformationsmanagementSystemaufrufLastVerweildauerGüte der AnpassungExogene VariableCachingLastParametersystemCodeEntscheidungstheorieKartesische KoordinatenSoftwareMessage-PassingWeb ServicesNichtlinearer OperatorMusterspracheBildschirmmaskeDatensatzBildschirmfensterDiskrepanzLastteilungCloud ComputingInformationsspeicherungSystemprogrammierungMereologieMultiplikationsoperatorMathematikInhalt <Mathematik>Computerunterstützte ÜbersetzungVariablePunktNebenbedingungWiderspruchsfreiheitDatenbankProzess <Informatik>Selbst organisierendes SystemBitInnerer PunktTelekommunikationMinkowski-Metriksinc-FunktionGanze FunktionMini-DiscClientRechter WinkelFunktionalWarteschlangeWhiteboardTermInformationSchlüsselverwaltungWeb-SeiteMailing-ListeInterface <Schaltung>Offene MengeFehlertoleranzEinfache GenauigkeitPerspektiveComputeranimation
13:06
Codierung <Programmierung>PartitionsfunktionProzess <Informatik>Streaming <Kommunikationstechnik>StapeldateiSystemprogrammierungWarteschlangep-BlockGebäude <Mathematik>SichtenkonzeptSchlussregelRelationale DatenbankModelltheorieDatensichtgerätWeb SiteWiderspruchsfreiheitRechnernetzVolumenvisualisierungClientMessage-PassingCodeDatensatzAuswahlaxiomObjekt <Kategorie>InformationProzess <Informatik>Streaming <Kommunikationstechnik>QuellcodeEinfache GenauigkeitFunktionalKartesische KoordinatenDifferenteEvoluteSchreiben <Datenverarbeitung>ProgrammbibliothekDistributionenraumMathematikNeuroinformatikOrdnung <Mathematik>PunktUmwandlungsenthalpieEin-AusgabeTelekommunikationDienst <Informatik>SystemprogrammierungStapeldateiSchnittmengeThumbnailGebäude <Mathematik>MeterDatenbankAlgorithmusProtokoll <Datenverarbeitungssystem>Funktion <Mathematik>WiderspruchsfreiheitWarteschlangeModelltheorieDebuggingWeb ServicesDeterminanteResultanteDateiformatSoftwareIdentifizierbarkeitEreignishorizontInformationsspeicherungSchlussregelAssoziativgesetzSynchronisierungDatensichtgerätMereologieApp <Programm>Formation <Mathematik>TeilbarkeitMultiplikationsoperatorBitDigitaltechnikIntegralInteraktives FernsehenKontrollstrukturDefaultInterface <Schaltung>Nichtlinearer OperatorCachingNebenbedingungKategorie <Mathematik>VolumenvisualisierungSichtenkonzeptDerivation <Algebra>BeanspruchungKreisflächeEntwurfsmusterComputeranimation
22:06
DatensichtgerätInformationProzess <Informatik>StapeldateiKomplex <Algebra>WiderspruchsfreiheitFehlermeldungRelationale DatenbankModelltheorieAbfrageVolumenPortscannerDatenbankTransaktionElement <Gruppentheorie>DigitaltechnikBeschreibungskomplexitätSystemplattformHypermediaQuellcodeSchlussregelGlobale OptimierungKartesische KoordinatenSchnittmengeSerielle SchnittstelleGanze FunktionEinsProzess <Informatik>VersionsverwaltungRelativitätstheorieMathematikServerAutomatische HandlungsplanungEreignishorizontTransaktionInformationQuellcodeInformation RetrievalBitClientProgrammfehlerAutorisierungDatensatzInhalt <Mathematik>Diskrete UntergruppeInverser LimesAbfrageSystemprogrammierungKonfiguration <Informatik>RechenbuchDigitaltechnikWiderspruchsfreiheitEvoluteStapeldateiWeb ServicesInformationsspeicherungFehlermeldungFigurierte ZahlNummernsystemDatensichtgerätKomplex <Algebra>CASE <Informatik>CachingFitnessfunktionGüte der AnpassungModelltheorieÄußere Algebra eines ModulsDienst <Informatik>AggregatzustandEinfügungsdämpfungOnline-KatalogMultiplikationsoperatorSchlussregelPrioritätswarteschlangeTranslation <Mathematik>MereologieREST <Informatik>InternetworkingRelationale DatenbankTeilbarkeitPortscannerMultiplikationSoftwareschwachstelleTabelleDatenbankEinfach zusammenhängender RaumEchtzeitsystemAuthentifikationSchlüsselverwaltungKontrollstrukturComputeranimation
31:06
TransaktionHypermediaSystemplattformQuellcodeAbstraktionsebeneSchiefe WahrscheinlichkeitsverteilungWiderspruchsfreiheitVersionsverwaltungBeschreibungskomplexitätPrognoseverfahrenProgrammfehlerAbfrageROM <Informatik>Funktion <Mathematik>TeilmengeSynchronisierungDatensichtgerätGasströmungUnternehmensarchitekturDienst <Informatik>WiderspruchsfreiheitSchreiben <Datenverarbeitung>BildschirmmaskeStrömungsrichtungSystemprogrammierungData MiningMereologieGrenzschichtablösungQuellcodeModelltheorieAbfrageInformationWarteschlangeKonditionszahlKontrollstrukturTabelleSichtenkonzeptTeilmengeNeuroinformatikStapeldateiSchnittmengeDatensatzKomplex <Algebra>SoundverarbeitungMultiplikationAttributierte GrammatikGanze FunktionDienst <Informatik>Prozess <Informatik>AlgorithmusAggregatzustandGleitendes MittelSystemplattformHalbleiterspeicherTermInformationsspeicherungDatenstrukturSchlussregelHypermediaCachingZahlenbereichMathematikDifferenteProgrammfehlerPortscannerIndexberechnungDigitaltechnikMAPLineare RegressionEreignishorizontSoftwareschwachstellePaarvergleichNormalvektorVersionsverwaltungAbstraktionsebeneFunktionalOrdnungsreduktionSchiefe WahrscheinlichkeitsverteilungEin-AusgabeInhalt <Mathematik>Computeranimation
40:06
COMXMLComputeranimation
Transkript: Englisch(automatisch erzeugt)
00:17
Welcome to my talk, I'm Jeremy and we're going to talk
00:21
about systems design, something I don't think we do enough of. Cool, the URL to my slides is apparently up there. Alright, for the past three years, I've worked at Kenna Security. Our application has changed a lot in that time.
00:41
It's had to evolve and become more distributed as our organization has grown and we've just been processing a larger volume of data. And we've learned a lot from that process of migrating and refactoring our system, but it's hard to find resources of how other
01:01
companies have gone through that. I'm leading one of these efforts to pull apart some pieces of our application, and it's going a bit like this. Not always, but sometimes. And I'm constantly afraid we're making these separated
01:21
but interconnected towers of responsibility that we think are encapsulated, but just aren't. A couple years back, I read this book and it's constantly amazing to me how much it translates to these kind and many other problems.
01:41
But all the examples in it are about code and how to rearrange code and how to make code resilient to change, which is great, but I want to know how that relates to the bigger picture, so start off with some code. For example, we have this library application
02:01
and this is a cataloger. I'm glad the syntax highlighting is visible up here, I was concerned. But yeah, what it's doing is it's taking a list of books from the internet and just creating a record of them in our system. And it's pretty much a good thing to get working,
02:23
but we've got a lot going on here. There's a lot of behavior packed into it, and if we want to improve it, identifying the roles and responsibilities isn't too difficult because there's some great tools and advice out there to help with this from the community, Sandy Metz, Katrina Owen,
02:43
they've made a lot of talks and have made this very common knowledge and very relatable and digestible. So we've got the squint test, for example, where we can see the shape and color of code to identify where something is reaching
03:03
into another class and knowing too much about how it behaves. Like we've got this API client that has some configuration going on in it. There's this parser, probably shouldn't be there,
03:21
and we're doing a lot of validation here as well. And these can all be pulled out and separated into small classes. And that's a much better design and it will encapsulate all of these roles. But what about this? We've taken that and put it into a worker,
03:42
just a background job, DSL, and it's not so obvious where we've abstracted away a lot of the configuration and how we're in queuing work and how all this relates together. It's not so easy to figure out where I can apply a lot of these design patterns,
04:01
a lot of these tools that I've learned, and it feels pretty bad. And then in the bigger picture of things, where do I even begin translating that knowledge of designing these things? If my system has a worker that's, I forgot to add an arrow.
04:23
If one of these workers is constantly having load issues because it's dropping due to disk space or it's running out of, just getting constantly overwhelmed, what are ways I can go about improving that? Where do I even start thinking about how to refactor it?
04:41
And my first instinct when I run into these kind of problems is to ask my good friend Tony. That's really his cat. But he and probably his cat can attest to the amount of stupid questions I've asked over the years. That's kind of the point. We've built up a vocabulary to talk about this and we've spent enough time talking about testing,
05:03
mocks, dependency injection, interfaces. We can identify a lot of good patterns from bad and we can communicate it. So why is there this discrepancy when we're talking about our system, about our application?
05:21
Suddenly once we move out of the code, we are throwing all our design tools out the window. But is this really so different a problem? We're still pretty much passing messages, very similar, and that's a question I've been obsessing on for the past year or so
05:41
and obviously it was on page three of Pooter. So design is about managing dependencies with your application and the people working on it. It's about being flexible and adaptable to change and having small, simple pieces that compose together.
06:01
So if I wanted to go and change the data store that my reporting service is querying over, how easy is it to do that? It's really not. It's difficult because these data stores, they're traditionally interspersed everywhere and the person who put it in
06:21
probably isn't even working at my company anymore. If I'm working at a startup, this was probably designed on a whiteboard and then immediately erased and lost forever. So as a new hire, I don't have any breadcrumbs to follow. And organizationally, I don't think we do well with this.
06:42
So I made a couple anti-patterns because I guess that's what I'm supposed to do. So they're mostly around communication. Talk about these things. If you're new, ask questions. Keep thinking about these topics and if you're established, be open to change.
07:02
Even if you don't think you have this problem, discussing these things and sharing knowledge is important. And then maybe document it, you know, maybe. More on that, don't have a few brains. Architect your system. We're in this together and there are many ways to reach a workable solution.
07:24
The social problems an organization has by exclusively having someone make these decisions are numerous and completely crushing to getting work done. Newer folks should be given the opportunity to learn
07:42
and possibly fail in the same way that more established engineers can because that's just how we learn. And not giving that chance means you're going to have engineers with hesitancy to make changes because they don't feel they know a certain part of the system.
08:02
And be aware of the trade-offs. Operationally, monetarily, talk through these costs and benefits of your decisions. And because we're not perfect, it's easy to make assumptions about the network. These aren't operations or systems problems.
08:21
These are things that we need to be conscious about when we're writing our code. When you're writing the code, it's there in your face. You're able to read it, but these things are actively abstracted away from your attention, but they have real consequences. We haven't talked much about what's going on here.
08:46
This is relying on a ton of things. It's relying on a lot of non-deterministic things about the network and the response we're getting back from the service we're talking to.
09:04
But what can I actually change about this? Oops, sorry. Not much. It's an argument list passed to a method. There's not really much we can do in terms of making this from the code perspective
09:21
more fault tolerant or more resilient to change or more flexible. We simply need to expand our toolkit past the code. These are the two best resources I know of about this. The one on the left is by Steen and Tannenbaum, and it's the academic end of distributed systems.
09:45
Not gonna lie, it's really dry, but it's packed full with a ton of details. It's focused on mainly making a single service like Elasticsearch that's providing a single function. Very highly available and resilient to faults
10:03
across many nodes and across a network. The one on the right is by Martin Klepman, and really gets into the weeds about the inner workings and trade-offs, a lot of the modern tools available to us from many cloud providers.
10:23
Lot of information in it. So the main takeaways from this book, at least to me, are that we want to hide communication as asynchronously as possible. You might not know how much ES nodes talk to each other
10:40
to health check to see if they're available, but it's a lot. And we wanna load balance effectively with queues. Our performance monitoring and queuing theory, these come into play a lot.
11:00
And don't be afraid to replicate data. Cache it locally to the client process. Bit more into that one, whenever we're calling outside the network, it's slow, expensive, and we're at risk to a lot of failures, a lot of faults in the network. Instead, we can store it locally into a variable.
11:22
This is a form of caching. We can create a record of it in our application. This is probably the most common, also a form of caching. Or we can simply put in a key store like Redis as a familiar read-through cache.
11:42
But what is not obvious about all these is when these caches expire, the invalidation question. Variables last for the entire process duration. If we have a 10-hour-long rescue job running, you're probably operating on stale data.
12:03
So yeah, a database entry could exist forever. It could be years since it's updated or referenced. And probably different from when you imported it in that external system. Only one of these is predictable.
12:22
And that's the read-through cache because we're explicitly telling it when it expires and when we have to rebuild it. That's the consistency cost. And it's one of the primary trade-offs we can lever and design around and optimize for
12:43
is determining our constraints and the SLAs we need to meet on how up-to-date and how real-time, how consistent our data needs to be. Sorry, that's a pretty dense slide.
13:05
And Kleppmann's book, it's got a lot of content in it. Some of the topics it goes into are encoding messages and protocols between services. RPC protocols like gRPC, JSON, other stuff like that.
13:26
It's got a lot of information on this. We're not talking about that. How to partition your databases. It has a lot of suggestions on algorithms to use.
13:40
And I would say the easiest one is to find a cardinal value. Find something that is unique to a set of data and use that. It's okay to use data stores for specific purposes. In fact, it's encouraged. Specializing some of these is more performant
14:03
than trying to make a Swiss Army knife aggregation in MySQL, just as batch your IO work into threadable workloads. And then if you have to change, or if something has changed,
14:21
you can stream those changes up to what you need to. And I wanted to spend a bit more time to mention events because there's a lot of misconception that stream processing is the same thing as event sourcing, and that's just false.
14:44
There's two ways to transfer information you can either transfer it by function, as seen by this nice arithmetic example, or by result. So if you're going to put all these into a log or a queue
15:04
they have different properties, which might be useful or detrimental to you, but you should be aware of that. The latter one is order-dependent. So this makes it very difficult to distribute work due to different computers have different clocks,
15:25
the time synchronization problem. Academics love that one. But this has the really awesome property that you have to just ensure that the last write wins. And if you can do that, you can ignore all the rest of the operations.
15:44
You can just be like, I've already written the final result of this. Just process the queue last and first out, and it's awesome. Likewise, on the left here we have a very cool associative property where we don't need to maintain sequential integrity.
16:06
We just need to make sure everything is processed so we can distribute many different jobs in parallel. They just all have to finish. But if we wanna recreate the state, we have to process all of them again.
16:23
It's a long way of saying that these are design patterns. They're choices to make. It's not a silver bullet. Make meaningful choices. And finally, in my opinion, the biggest takeaway from this is determining systems of record
16:44
and derive data in your system. Poorly represented by me here as functions, as you can see, function F relies on the output of G that relies on the output of H and so on and so forth.
17:01
We have to process all of them in order to process the final one. And they have to be processed in order. So if I make a change to how function H behaves, I have to reprocess all the other sequential work. Alternatively, if, ooh, and that's looking great.
17:25
Alternatively, if we pretty much have all the functions pull from a single source of truth, a single system of record, if we can make changes,
17:41
they're not dependent on each other. It's basically the difference between having a logged meter violation and a composable object. This is a code smell and a place for optimizing your system so look out for it and yeah, full circle.
18:02
By thinking about our system in these terms, we can tease apart where some dependencies and design for what they depend on, which is the data. Some rules of thumb, use the actor model. Communicate through message queues as much as possible.
18:20
Make your work asynchronous. Make workers retryable. Have the work they do be idempotent. There's a reason this is in the 12-factor app. It's very strong. And pre-compute as much work as possible. If you know what you need to display to the user,
18:41
do it out of band from when you have to display it. Let's see, yep. Identify where data is derived in your application. This gives you some nice seams to make asynchronous.
19:01
Determine the consistency costs and the constraints in your system. Pay for what you need to be real-time and relax the constraint wherever you can. That will allow you to keep the interactions between services in your system simple.
19:22
And with that, I wanna spend the rest of the time revisiting our library example, making some building blocks, and showing some of the evolution of our system at Kenna. So, building blocks. Yep. The first major piece here is the worker.
19:42
This is how we process asynchronously. If we're transparent with failures, if we're making these things retrivable, we can just kill the process when it faults and push the work back onto the queue and let it reprocess.
20:03
Workers pick up the work from queues, pick a format for messages, and try to optimize simply by the queuing behavior. This lets you avoid redundant work. Materialize your data into a view. Pre-compute what you can, invalidate caches in a very predictable manner.
20:26
Have artifacts built from expensive computations so you can circuit break and default to the value from yesterday if the job failed today.
20:40
And these give you very simple interfaces between the parts of your application. So, let's look at some universal behavior that I think most applications have, like interfacing with an external API, or maintaining relational models across the network.
21:00
Deriving data points through business rules and performing computations on our data set. And finally, high read, low write. Spoiler, that problem is the one you want tooling for. Don't reinvent the wheel, guys.
21:24
Say we want to display reviews from Goodreads or an external service. How can we go about sourcing these into our application for the user? We can simply embed this directly into the front end.
21:43
This gives us as hard a consistency as possible because we're literally displaying what is in the external service, but we're relying on the network for every render we're making. But if we put that into a background,
22:01
or into the back end, we can have a read-through cache, and we're making a single request. We still have a real-time render, but still time request on render, but it's an improvement. And if we know what we'll need, we can simply use a worker
22:23
and process this in the background. We can make this query asynchronous and retrieval for information if that fails. And we can figure out what we need to display to the user and push that into a highly available data store
22:42
so it's not actually hitting our server. However, it's hard to determine what changes because not all APIs give you a change set. Hey, here's everything that changed in the last day. Doesn't happen. We can't pull in the entire Goodreads review data set.
23:04
It's just simply too much. So this isn't always applicable. But what if we want to make updates back to the external service? What if the user wants to add a review? This becomes a pretty tricky problem.
23:23
Simply because we don't manage what we're keeping consistent. One option would be to use a client consistency model and display the new content exclusively to their JavaScript session. The user thinks that they have updated or added a review to the Goodreads system,
23:44
but we're processing that at our discretion. And we can batch these into one thing if there's an API request limit. So there's a lot of good here. But it's obviously given us a bit of complexity.
24:01
And we don't handle authentication. Ah, there's a bug. What the hell? Oh, sorry. We don't handle authentication errors well. And we're technically lying to the user about what we've done in our system.
24:21
Which isn't always the best thing to do, but sometimes it's better than the alternative. Another approach would be to update our internal state and then validate the cache with a priority queue. This is most often what we're doing at Kenna. It's worked very well. It'll work for like 90% of your use cases.
24:44
And we can still process things asynchronously and retry on those. You get a lot of benefits working with this, but you're also making trade-offs. So figure out which ones you can tolerate.
25:02
So for relational models, how do we maintain relational records when the data is in separate places? Say we want to add a book and an author and display that to the user from a background job like our cataloger.
25:20
How do we maintain transactions and joins across multiple tables? These get really complicated by service-oriented architectures. In the monolith, this is fairly straightforward. We can wrap each row into a single insert,
25:41
each insertion into a single transaction. It just works. It's straightforward, it's simple, it's very reliable, and you have guarantees of consistency from your data store. If you're using a relational database, I guess.
26:01
But you're given a hard upper bound on the amount of data you can have in one place. And if you're approaching that, you will get significant performances, much like we did. So we partitioned our database by a cardinal value,
26:20
clients in our case, because we process data, clients specifically, and this fit into how our role-based access scheme was working anyways. And we keep a lot of the good qualities of having a relational database store all this. Transactions and joins are still easy to do.
26:43
But it's very hard to retrofit. ActiveRecord doesn't handle database connections for you. They're working on it in the new Rails, but it's not there. We can't table scan across multiple databases,
27:01
and you're gonna end up with hotspots like a client that has a million vulnerabilities in our case, but it's not perfect. The first suggestion the internet will tell you to do is to wrap everything in an API. Divide everything by foreign key,
27:20
and give it a RESTful interface. It's the simplest translation you can get from how we store information in a table, but joins are nonexistent, transactions aren't guaranteed between two separate tables, and you're simply relying on many more services
27:45
to have availability to display information to the user. I think there's a lot more bad than good there, so then the internet will tell you
28:01
to keep transactions in lockstep with each other. You want to have a priority, or not a priority queue, you wanna have a event stream, or event log, and then transactionally update things, and when one fails, you just roll everything back.
28:22
In blog terminology, this is a saga, and you get much better consistency, or you get much better, I'm trying to say, you can ensure that a lot more things
28:40
are the way they should be, simply. But we've coupled our distributed processing together, where the one data store is now relying on the other one to finish processing before it can begin, and we're still left without an ability to join between these two services.
29:05
So, still not great, but what if we simply didn't care? You're not gonna have perfect data in your system, so plan for it being bad, choose what you're going to display, invalidate the relations you don't have,
29:22
and version the ones that you do. Whenever anything changes, have a worker serialize that change set out. Pre-compute what you're giving to the user or another service. Materialize the joins and optimize subsequent batch calculations.
29:41
Circuit break, sorry. Problems you get with this is that you have now given yourself
30:01
something to keep consistent with the rest of your services. And it's a bit more complex. This is the evolution we've seen our system take as we've hit the different limits of the data stores we're using.
30:23
Plan for the problem size that you're experiencing. Don't over-optimize. Use the simplest solution, but anticipate it might have to change, it might have to grow, and you might have to go into something that's a bit more complex.
30:41
So determine where those parts of your application, where those exist in your application. So I guess I should speed up.
31:02
So how do we describe business rules in our system? Say we wanna source new records from multiple social media platforms. The naive approach we can have is find or create. We can have a simple transaction-based
31:21
create, record creation, and we can lock on records. So we're eliminating some race conditions, but there's just the inherent race condition of processing this way. We lose information about the sources. We're actively overwriting information going with this.
31:46
CRDTs are cool. I'm not gonna talk a lot about it. It's a data structure that knows how to resolve writes. Martin Kleppman has a cool talk on this. I recommend watching it.
32:01
In reality, we probably would want to save data as close to the source we're receiving it from as possible. We'll have as clean a form of the data, and we can normalize it together
32:22
into a way our system wants to use it. These don't have to be separate services. It's just drawn that way, and they usually aren't. These are pretty much separate models or tables, and it's easier to debug inconsistencies with the information you're getting from the external API
32:43
when you're not having to use the external API, but you have to maintain consistency with the source. You have different abstraction levels on what these records mean. But what are we really doing here?
33:01
We're, if modeled through an event log, this is really just a derived view of different parts. This is really just a derived view of different records. We can push out different versions
33:21
of our business rule algorithm and compare their effectiveness. We can version these, and suddenly we have a very powerful tool to compare work we've done, I guess. I'm not sure what I was saying.
33:44
So let's talk about computations. Effectively, how do we query? How do we compute? How do we make sense of our data
34:01
in a way that we can provide value to our customers? The first thing you should do is publish an artifact. If you're updating an attribute on a record,
34:21
you're doing it wrong. This is solely derived data from an expensive computation. If you have a distinct artifact from it, you will be able to just circuit break and use the one that exists
34:43
if your current computation fails. You can also detect anomalies by running some sanity checks before pushing it live. You can compare improvements and regressions between your computations. Very simply, if they are in-files,
35:04
there's not much bad about doing this. And the first improvement we can have over a table scan of our entire dataset is to batch it up, just find in batches. It's easy.
35:21
Active record makes this kind of thing very memory efficient. You're still IO bound here because of table size, run into hotspots, all that stuff. And if we've partitioned our database,
35:40
this rolls very nicely into this. We're less IO bound, but we still have the problem with hotspots and SKUs.
36:03
So the term for this, or the algorithm for this kind of work is MapReduce. If we want to include more context, more data into a job,
36:20
we can just add a simple step that translates or joins data into our computations. My thinking on this is, instead of having a data, or a step in our processing join this data together,
36:42
why don't we simply materialize a view and batch from that? We're reusing a seam in our system. We are operating on exactly what we're displaying to the user. If we have it well versioned, is versioned,
37:02
depending on the data store you're using for this, it might not be optimized for subset querying. So one thing we have done is to use fact tables and change sets. We publish a CSV entry for changes and send that up to an optimized data store
37:22
like BigQuery. It's kind of hard to debug when the numbers don't add up and you're effectively monitoring the entire state of your data, the entire data set you have. So it has some complexities there.
37:40
Tooling, so. This is pretty much what we've done in our system. It actually looks fairly similar to this terrible drawing of mine. We have two primary job queues and what we need,
38:01
the one on the left is what we need to process and normalize into how we manage data or how we, pretty much our business platform. The one on the right identifies what we're actively serializing into Elasticsearch and what we're providing to our users.
38:22
Subsequently, it's two primary MapReduce functions. We're importing data and normalizing it together. We're reducing it into a normalized set of vulnerabilities in our sense and then we are pretty much fanning out and then putting it into search indices
38:43
and documents that are readily or quickly available. As you can see, most of this is completely done asynchronously because we are putting everything into queues and pulling from it with workers
39:01
and serializing stuff out. The question often here becomes how do we compose these data sets together and at what points and if you want more information on that, better numbers, my friend and colleague, Molly Strooby,
39:21
is having a fantastic talk on this tomorrow called Cache is King, I highly recommend it, it's awesome. Because in the end, this is really about simplicity. It's how you can reason about your system and it's how you can collaborate with others on it.
39:40
It's all the content I got. Thank you so much for listening through it. For more on this topic, these are some of my favorite resources. Yeah, thank you so much.
40:02
Here's some contact information.
Empfehlungen
Serie mit 14 Medien