Bestand wählen
Merken

You're Building a Distributed System!

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
and
11 becomes the 1 you we get excellent I thank you for coming out of the the last talk of day around the home stretch has less said and I and I feel like it's sort of my responsibility to make sure that all of your brains are completely full and potentially there may be some buffer overflows today so we'll see what happens and but thank you again for the coming out let's talk so a mere day talk to you about state really want talked about state this seems to haunt me everywhere I go this this is
about representing state in object-oriented fashion the or transforming state in a functional style this is about how the structure and means of transferring state across the wire thank fully we have to like PI decisions about the long term durability and persistence of state like answer so there really mean you're making decisions about how state gets represented and transfers throughout your and replication the like with a query parameter the especially if you're making decisions about how your state gets represented the highest state gets persisted for a long-term durable storage I'm here today to tell you that your building a distributed system the my name is mike pack I'm not an expert in distributed systems in I happen to learn a good bit about what I'll be talking about today and the cost of trying to take a product to market in Africa it really wasn't until I did manage to step back from taking a product to market until I realize with crap um when we when we're building stateful applications in the browser but with Amber which is highly stateful embraces states what we're really doing is building a distributed system the this also try all the trying to prove that a showcase that to you today becomes you know the real takeaways I want and I want you to leave this room with our sort of a holistic more holistic understanding of a state in different types of state but I want to again convince you that when you're building applications with Amber and state fluctuations in general that you really building initiatives system a need to think about you're your applications and the problem that you're up against in terms of distributed systems this is not really a talk to show you solutions per say the building distributed systems although of course that sort of baked into the discussion so I work
for a product consulting company out of Denver Colorado called modes and we're approach to in 2014 by a gentleman named cop hammer or Katie as we call them and keep you want us to rebuild some existing software that he had been market in Ugandan particularly wants to rebuild this in modern technology that accommodated so the problem that he had with his initial version of this product member was a natural choice and and you know offline was 1 of his major pain points and so will be focusing a good bit offline today and is a picture of keepyour myself in Jinja Uganda this is the source of the Nile River did is a beautiful place but just amazing I would highly recommend if you're going to Uganda to visit ginger so this is the Nile River behind us in this photograph work with an array of people winery people across Africa all the way from West Africa in Nigeria and Ghana through Uganda and all the way up and you know east to Kenya so I'm regularly working with these folks and is the some of the fine folks that I work with in Uganda a travel therefore for 2 weeks and this past August and I happen to have this happen to line up in this fashion you sort of serendipitous and this is it i love this photograph deserted you know some of kind people ever had the opportunity to work with are they big deal with some interesting problem that we all have to think about in most cases while building applications here in the US but within the 2 weeks that I was in Uganda the the network the Internet went down about 5 times and the power went out about 3 times they were between 1 in 6 hours at a time so these are real issues for them and I'm sure this is not anything new or and this is you know pretty obvious that they have these problems but they're they lived in the next part reason I went to Uganda was to experience that and to live in their environments really embrace how they experience these problems and of course some of the technologies that they're using some the computers at the using adjusted to talk to people who are using of were using his existing software so that we could you touch on the right pane points actually have some of these T-shirts with me today it seems like a number you snatch them off of this slide tables that's also have a few more here with me on stage so come find me after this talk and that that's in a couple of ones as well the so I mentioned I'm here today to talk about states and there's 3 types of state that I want to introduce some of this might be familiar with all talk 1st about 2 types then introduce a 3rd these 1st 2 types are ephemeral states and permanent state there is a mentioned I'll be introducing a 3rd type of thing so let's break each of these down individual and so ephemeral state what is premise that you of ephemeral
state as being local to the user's browser this is typically cleared on page refresh and this is generally not persisted to a server it's generally not persisted up example the thermal state might be a component that you have an ember our you haven't is open property when you click on that component you would expect that that component adds this is open class and then opens the component is a basic did when when this is open you refresh the page you expect that DIV on page refresh but comes back in its collapsed state that's the premise that current state is state that we can generally think of as living on a server somewhere so ephemeral state you can think of as state in your browser permanent state you can think of as state on your server somewhere and this 1 and a 1 important characteristic of state is that it's written to disk to and that'll become increasingly important as we talk about this and when Billy off-line applications there is an example of some current state that you might create other factors actually showcasing both ephemeral and permanent state so we have we call this stuff stored up create record and in our case it would be a patient record and that's creating some ephemeral state in the browser in memory In Amber in the DS store so we might call save on that record and that will transform our ephemeral state into permanent state on the server so let's look at how this how these 2 tied to state fit into your repetition to architecture the so have your application here at the top and within your application you have some ephemeral status might be like our is open component you might be using and data I'm going to be a but I'll be talking about member data in a from examples so the number down you create this patient record you creating more from state within Amida the all this is of course happening in the browser and then we call save on that record we are converting that ephemeral state into permanent state on the server so all that's happening over HTTP servers of hopefully consistent back but that's over HTTP and out then through the server and of course writing that States you disk somewhere that's important to prevent data loss to writing at current status really I won't talk about today is what happens when we go offline so I use this dotted line here to denote but a partition offline connections so that's what we talk about today is how we accommodate this particular scenario but I think use that 3rd
type of state that I mentioned this type of state is semi permanent the and we need to introduce this in order to solve this problem of offline applications and making those applications fully functional so some characteristics of semi pro state would include the fact that lives in the browser and and it's transitory sort by definition it's not necessarily state that you would expect it to be stored permanently uh for a long period time and it's written to disk much like permits or seeking think of semi-permanent state as state that you are restoring as current state more or less in the browser example of this permanent state might be using local storage so local storage here if you set an item in local storage you would expect that when you refresh the browser that States still exists and you can pull that state out of local storage a use local search here's an example but what should be talking about index TV for the rest of this talk the so again this is the problem that we're solving or we have an application that has gone offline when the network is you no longer accessible what we have to add in a semi-permanent state in order to accommodate this particular problem but so this in this example again we're using index the b here and that our 2nd box there is is the best and that is what accommodates the fact that we can write data that will not be lost so were writing data in semi state but to index the now I wanna mention that you don't necessarily need to add a from state of course in applications that don't need to accommodate offline scenarios but in order to make your software work in offline environment of course you have to have something in which you persist state otherwise if all your state was ephemeral and the you know the user's browser crashes or the machinery restarts there and the abundance of of things can happen at this point we wanna ensure that stated not lost so we need to write that state to disk locally in the user's browser but is 1 important aspect of when you write state to semi-permanent store that you might not experience if you don't have index in your applications and that is the problem of synchronization the limit just quickly go through an example what I mean by synchronization to say we have a an number application the browser and we've created a new object in member data and then objects might be a conference object with the name of amber color and this was a mistake from the user types Amber called of course they mean ambit of yeah the the some of the old time that Amber fans will get this joke uh so when we tell the server that wanna persistent state to permanent store that state will then the server will return us a 200 response and the services OK i've written out to disk you can carry on with the business I guarantee you that state has been stored in long-term permanent durable storage so now both the browser and server have Bhopal so the user says 0 yeah of course I you know Ms. type that actually and and and recall so the brow the the user types it and therefore built it a and now the browser and the server user needs to tell the server OK you know I've I've store that's the energy the story that's the but this trigger time something happens we go offline maybe the user's browser crashes maybe there is an abundance of things that can happen as I mentioned even some at the lexical level failures can happen it can happen at a system level it can happen application level the number of some of ways in which we can get into the state but the point is that we can no longer communicate with the server so now the browser has and and the server has amber cough so these are these 2 pieces of data between the browser and server need to be synchronized we need to get to a consistent state in which they both contain the same data this is not currently the case as a number thing that can arise in this particular scenario there could be conflicts in which the browser has some state that the server refuses to accept it or there's problems with merging that data so we can we have conflicts there's a number of things that can go wrong in this particular scenario the the synchronization really needs to happen between 3 pieces of our application we have are ephemeral state again that's number 1 in this diagram number 2 is our permanent state number 3 is of course current state when were synchronizing state and we store that state a browser we 1st of course creation ephemeral state which then needs to be synchronized with our semi permanent state and then that needs to be synchronized with a permanent state now if you're building and repetitions in the US you like you don't need to have this 74 instead slightly likely not factor for you he simply just don't need it if your application doesn't need to function offline but there's an interesting characteristics that happen when you add semi-permanent state to your application is not necessarily to accommodate the offline scenario there some other nice benefits from this as well and that you can read by adding this year online applications so those benefits might be minimizing the the loss of data since you're writing that data to disk immediately in the browser you have guarantees that if something happens terribly at that time that that state has been persisted permanently or semi formally in the browser and that you can carry on with the business so we're minimizing data loss and then we can synchronize sort of behind the scenes are
asynchronously we can synchronize between number 2 and number 3 here between a semipermanent on a permit story behind the scenes the the the other aspect of this is that it allows us to create interfaces with your latency so because were writing to disk immediately we can resolve our promises very quickly to increase interfaces with 0 latency and then again behind the scenes that data synchronize between our semi-permanent store a promise so you're and repetitious probably some more on this scale of robustness you have probably some components of this if you don't have a server and you simply storing data in index the being that's all your application does then this is a really apply so much to you but I would assume that a number of you have some server somewhere story some of your state but particularly when you have parity between your state in the browser and the state on the server you have to deal with this problem of synchronization this is common in distributed systems in particular distributed so a simple this diagram a little bit and just say that we have a browser and the server and also generalize this a little bit and say that we just have computer a and computer B this edition of the system itself this distributed you have nodes in different places all running on different machines but of course it services and can have more than 2 nodes they can have 3 nodes an additional system can have and those of course where your and replication is 1 or more of these nodes in this system and probably you have a server and so maybe 1 or more of these nodes by is your server or at any number of servers that might be communicating with so if we knew synchronized between N number of nodes in a distributed system I we think about this problem what have we think about the synchronization in the challenges of synchronization what there is a way there is an established way this is is a lot of research that's been done in this domain but will be talking about today is is with capped where phrase this predictor problem is with capped gap here now you're like me you hear the word theorem that evokes nightmares from from your early algebra days I will try to not evoke some of those nightmares for you here today so let's break out cat means and how it relates to ERM replication and should systems and state the cap itself stand for consistency availability and partition tolerance you may have heard me use the word partition a little bit earlier so the breaking point is that consistency effectively means that all nodes in the distribu system have the same that we are in a consistent state so when I look like in a consistent state is we have the browser and the server and they both have imbricata it inconsistent state of course the browser has different data the server so the browser has calls and the server has in before the again I use this starting line here to denote a petitioner offline state so simply means that the software is fully functional this is not the theoretical definition of availability is my eyes simplified definition but actually it works the so an available system when we're all were more online Of course the system is fully functional as probably like most of you repetitions you're able to the is fully functional because you're able to communicate with the server when you need to now we get into a disconnected state and offline state by this definition the software is unavailable probably for most Member application is not with any type of semi current state yeah but when we get into a unique experience the network partition when we experience an offline state the software is no longer fully functional for probably most of the year and applications so this is unavailable is not really very usable that this is not really what we're going for the year are going to accept the fact the software breaks when the network partitions this is why we need to add semi-permanent state with index TV In order to make the software fully functional again so we add index the B and instead of assuming that we're going to write that data to a server somewhere we instead write that to index the B so as long as assuming that index the the can do everything on our server can do and that's not a caveat to be aware of we have now created of peace after that is fully functional while offline so we have now reached availability again and now partition tolerance in practice as I mentioned partition is when 2 nodes in a distributed system disconnect for some reason there's some failure that means that those 2 nodes can no longer communicate but in practice what this really means is you have a choice between being consistent or being available you can choose 1 or the other and that is actually what captive was saying this is a conjecture I'm originally and was proven that as fear the and so partition tolerance is based off of the
principle Oh no I'm kidding partition I have the Gapminder just so that you know but the 3rd but is always based off of the principle that computers will fail they will inevitably fail so we design a robust distributed
systems we designed for this value we designed for the failure 1st and then later in the ability 4 of a successful case a or a and the normal happy that but again sort of in practice this means the choice between being consistent or available the so here's 1 of our choices we haven't unavailable system but this is a consistent system now the rather has ever come from the server has ever concerts therefore consistent but since then available it's not fully functional and sometimes it's OK i'd i'd actually posit that it's usually OK to just stop here and say let's disable all form field that's a stable all buttons and our software will not be fully functional while the offline partition state but that's OK because making the system work in offline state adding additional layer really add a lot of complexity so it's actually OK I'm not trying to tell you that you should always build your systems with some type of semi-permanent state effect and saying that you may be would wanna consider just stopping here that's not really what we're going for when we're building off-line applications what we really want is available system and if we want the system to remain available we have to accept the fact that it will be inconsistent and we will need to resolve those inconsistencies in time this is an idea called eventual consistency the course to make that available to make a fully functional we add index B and we write and experience that we assume that I will eventually get consistent and will eventually have become the same data at the server contains in the system will figure it out and so really going off line means the working with some of the tools in your browser and we've heard about a number of these tools today so we've heard about service workers that allow us to cash assets and work with data we will use service workers to work with the data with data in particular and in fact we actually used at cash and just after browser compatibility but it's really just working with service workers in working with index the B which is the document database that probably should define this earlier but the document database in your browser but going off line is sort of the easy part write caching your assets caching jobs style sheets in images this sort of easy part in your browser can just pull them from cash the real challenge in building off-line applications is synchronization so we come back on line and now it's time for us to synchronize without premised so I think that we've written to index the B as semi-permanent state we now need to propagate that state to the server somehow so this is actually the challenge that were up against when building off-line applications is the fact that we synchronize between the semi arch semi-permanent state within within within the next maybe an hour current state on the server somewhere as this in great tools that we already have but to solve some of these problems the 1st 2 I wanna mention is couched DB and pouch else a group fired days into this as well because be is a document database that has a synchronization protocol that is exposed part of public API which is sort of the difference between couch Debian maybe some other document the power CV sits on top of index the and rights to index the been deals with the synchronization with couch the behind the scenes so the combination of Council be impartially is what I was which handles the synchronization problem for more or less but of course as I mentioned earlier we still have to accommodate for the fact that there may be merged complex of some sort the when we do come back line to catch the B provides us with merge functions which are terribly complex and difficult to manage over time we and fire based on is doing this in an offline 1st manner so 5 is a great tool if you are able to drop this into your application you sort of get the offline functionality of the box they don't actually have yet needed the driver for the Web does not have this ability to write to use any permanent state and synchronize that with permanent state on the server yet that the where the driver does not have this but the library for native does so this is coming to the web the it is dropped by base in your application you use their libraries are able to just 5 is handled all for you this is not really an option for us are in our case we have some prerequisites on existing databases on a relational database and so in that case we need to leverage tools like orbit JS and we sort of went through the gamut of what to was the most appropriate we or JS is effectively a FIFO queue that when you tell it to say that I would make some type of transformation it will then she what those changes and play them on all the different stores the set up such that you're going to have 1 of your stories being your API and 1 of your stories being indexed the and it will handle processing that FIFO queue and working through all those transformations so this is a tool that we end up on and this is you know what you need to use if you're not able to use an offline 1st technology like fire like couch to be without thinking as fun not very quickly to see our cities stand for conflict-free replicated data types and are sort of the future of avoiding merge conflicts if your offline for let's say offline for a day or so possibly could be offline for a week at a time your offline and you come back online there's all types of things that need to be merged and need to be synchronized and if there's conflict between 0 that's actually what the harder parts about building distributed systems is managing those conflicts to society keys are designed to avoid all conflicts any all conflicts there is some mathematical principles that allow you to effectively like a set union is an example of this it allows that data tumor without you having ever write merge functions the year the the sort of the future of this and if you can represent your data as a CRT t you can go offline indefinitely and that data is guaranteed to converge into merge properly and to create the correct value in a predictable way the so certain
conclusion here why is that why am I talking about this today so
course this allows us to go offline but also allows us to minimize data loss as a dimension because the right immediately to index the and allows us to create interfaces with 0 latency because were writing to index the the which resolves are promises immediately or more or less immediately the through interface was 0 latency imagine the user experiences so imagine how much better that user experiences when we can write data immediatly and result promises since it creates effectively a desktop application and what is nice about on applications right is why people like native applications the snappy you generally don't have to wait so for data unless know you're talking unit the ice is not actually that dissimilar from what we're dealing with the browser that is all work in progress and if you're going to build off-line applications you're going to think about using offline 1st approach you need to know that this is all still work in progress and that you will absolutely be blazing your own trail is still work in progress in academia this still work in progress in the industry so these are unsolved problems and they need you to be involved in creating solutions so but that thank you
you
Videokonferenz
Physikalisches System
Puffer <Netzplantechnik>
Code
Endogene Variable
Quick-Sort
Aggregatzustand
Subtraktion
Bit
Punkt
Browser
Versionsverwaltung
Zahlenbereich
Wärmeübergang
Kartesische Koordinaten
Aggregatzustand
Term
Internetworking
Eins
Message-Passing
Digitale Photographie
Software
Objektorientierte Programmiersprache
Datenreplikation
Datentyp
Permanente
Kontrollstruktur
Speicher <Informatik>
Datenstruktur
Drei
Leistung <Physik>
ATM
Parametersystem
Expertensystem
Datennetz
Medizinische Informatik
Namensraum
Fluktuation <Physik>
Default
Güte der Anpassung
Abfrage
Quellcode
Physikalisches System
Biprodukt
Menge
Quick-Sort
Entscheidungstheorie
Rechenschieber
Arithmetisches Mittel
Rechter Winkel
ATM
Mereologie
Programmierumgebung
Tabelle <Informatik>
Aggregatzustand
Einfügungsdämpfung
Punkt
Browser
Web-Applikation
Versionsverwaltung
Schreiben <Datenverarbeitung>
Kartesische Koordinaten
Aggregatzustand
Synchronisierung
Homepage
Übergang
Komponente <Software>
Permanente
Gerade
Datennetz
Kategorie <Mathematik>
Stellenring
Strömungsrichtung
Frequenz
Teilbarkeit
Dienst <Informatik>
Datenfeld
Automatische Indexierung
Festspeicher
Server
Charakteristisches Polynom
Ordnung <Mathematik>
Programmierumgebung
Fitnessfunktion
Aggregatzustand
Server
Quader
Klasse <Mathematik>
Zahlenbereich
Systemzusammenbruch
ROM <Informatik>
Division
Homepage
Demoszene <Programmierung>
Datensatz
Fächer <Mathematik>
Software
Mini-Disc
Datentyp
Endogene Variable
Inverser Limes
Speicher <Informatik>
Einfach zusammenhängender Raum
Default
sinc-Funktion
Browser
Permanente
Physikalisches System
Partitionsfunktion
Quick-Sort
Objekt <Kategorie>
Energiedichte
Diagramm
Komponente <Software>
Kantenfärbung
Mini-Disc
Unternehmensarchitektur
Bit
Konfiguration <Informatik>
Punkt
Browser
Computer
Kartesische Koordinaten
Synchronisierung
Komponente <Software>
Theorem
Datenreplikation
Speicherabzug
Kontrollstruktur
Gerade
Auswahlaxiom
Schnittstelle
Automatische Indexierung
Zentrische Streckung
Datennetz
Systemaufruf
Debugging
Strömungsrichtung
Widerspruchsfreiheit
Kugelkappe
Arithmetisches Mittel
Software
Dienst <Informatik>
Gruppenkeim
COM
Gerade Zahl
Automatische Indexierung
ATM
Server
Ordnung <Mathematik>
Computerunterstützte Übersetzung
Aggregatzustand
Algebraisches Modell
Server
Subtraktion
Systemverwaltung
Zahlenbereich
Kugelkappe
ROM <Informatik>
Physikalische Theorie
Demoszene <Programmierung>
Physikalisches System
Virtuelle Maschine
Computer Select
Knotenmenge
Domain-Name
Software
Adressraum
Mini-Disc
Datentyp
Speicher <Informatik>
Widerspruchsfreiheit
Hardware
Einfach zusammenhängender Raum
Fehlermeldung
Medizinische Informatik
Browser
Physikalisches System
Permanente
Partitionsfunktion
Auswahlaxiom
Diagramm
Touchscreen
Komponente <Software>
Wort <Informatik>
Mini-Disc
Bitrate
Konfiguration <Informatik>
Prozess <Physik>
Browser
Gruppenkeim
Versionsverwaltung
Computer
Kartesische Koordinaten
Komplex <Algebra>
Farbverwaltungssystem
Synchronisierung
Komponente <Software>
Prozess <Informatik>
Speicherabzug
Permanente
Gerade
Auswahlaxiom
Automatische Indexierung
Lineares Funktional
Datenhaltung
Debugging
Strömungsrichtung
Widerspruchsfreiheit
Konfiguration <Informatik>
Software
Dienst <Informatik>
Orbit <Mathematik>
Datenfeld
Gruppenkeim
Menge
Automatische Indexierung
Rechter Winkel
ATM
Server
Schlüsselverwaltung
Aggregatzustand
Server
Subtraktion
Quader
Systemverwaltung
Schaltnetz
Zahlenbereich
Transformation <Mathematik>
ROM <Informatik>
Demoszene <Programmierung>
Physikalisches System
Computer Select
Benutzerbeteiligung
Bildschirmmaske
Reelle Zahl
Software
Adressraum
Datentyp
Programmbibliothek
Elektronischer Fingerabdruck
Warteschlange
Speicher <Informatik>
Bildgebendes Verfahren
Widerspruchsfreiheit
Hardware
Leistung <Physik>
Soundverarbeitung
Relationale Datenbank
Fehlermeldung
Protokoll <Datenverarbeitungssystem>
Browser
Orbit <Mathematik>
Permanente
Physikalisches System
Partitionsfunktion
Quick-Sort
Druckertreiber
Touchscreen
Mereologie
Mini-Disc
Bitrate
Videokonferenz
Einfügungsdämpfung
Weg <Topologie>
Einheit <Mathematik>
Extremwert
Arithmetische Folge
Automatische Indexierung
Hausdorff-Dimension
Browser
Ähnlichkeitsgeometrie
Kartesische Koordinaten
Maschinensprache
Schnittstelle
Videokonferenz
Dienst <Informatik>
Ereignishorizont

Metadaten

Formale Metadaten

Titel You're Building a Distributed System!
Serientitel Ember Conf 2016
Autor Pack, Mike
Lizenz CC-Namensnennung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben.
DOI 10.5446/34722
Herausgeber Confreaks, LLC
Erscheinungsjahr 2016
Sprache Englisch

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Whether you know it or not, you're building a distributed system. This becomes painfully obvious when taking Ember apps into the developing world. Flaky network connections pose real challenges for developers building stateful applications in the browser. This talk is about my trials and tribulations taking Ember offline in Africa. We’ll look at the tools necessary to venture offline, how they apply to Ember specifically, and a little bit of theory to drive home the hard facts about how much fun you’ll have building a distributed system!

Ähnliche Filme

Loading...
Feedback