Merken

Amanda: A New Generation of Distributed Services Framework

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
check
the morning everybody all the talking about and are distributions
platform so I will be shown in yellow pretty pictures I just hope that the talk a little to the amazing work everybody else is doing but we think about myself 1st I have
been suffer developedin discussions about thousand and to work with Python since 2009 love services and using this plugin-based slight units us the monitoring of the various formal 3 morning and had to drill the intentional but also for life of quiet was a great opportunity on the so about
infrastructure to you then NBC
of been working as a status of development and integration textual advertising institutions these are of the movies that we're working on recently
and which you do this across these sites with multiple fully
integrated cross-site pipeline was make sure that the data flows from 1 side to the other depending on what the departments are where they work so I guess
not everybody here might be specific the familiar with what these effects also this a pretty good quotes from more from Wikipedia from which comes down to everything that is under extensive dangerous or with actor during filming as we're trying avoided by this couple of actual images of the work that we do is going to be a bit better so this is
a shot from from what was years
from clients and is actually we get to see everything that you see there in the background to the fake the same thing here this is a
short from Godzilla 1 of the latest movies as anything it is what we got in and is the actual work was done to if you look like and you can see that the guys and the thank got replaces
CG characters 1st of all the
things that used so the the work of a lot of assets where as it is something like you know a creature architecture or whatever else that we need at this stage we actually
shouldn't flows through the whole system to do that of course artist there's there's a bit of his magic once this is done creation we call a daily which is a short introduction show the work that he has been doing that can then be reviewed by supervisors once that is done you can you can approve acid and of the cannibal's that some comments and things from there was a progress goes through releasing stage where a lot of things happen and we actually create directories where we store our data we added some item meta-data about processes or make sure that everything flows into the next apartment so you for example we got our modeling team which for example phrases that an actual
character and then these some taxes and and well we release that you make sure that we all take all the dependencies to make sure that we notify all different orders that new things have been released and estimators that using any data that we have for the different sites so cause if people above
all things in mind I'm doing this there's not 1 artist working about 16 hundred working and released thousands of years of acids today so as to that in mind and ever-changing status 1 day might be quiet and next thing you might you have completed the researcher would traded and he's delivered in the problem which was the the level of different sources that we
use coming from the database the body of the ice storage all different locations and the use by in-house tools that we have been writing a third-party applications that are destined to work with and of course that will allow the actual mole environment we don't work once environment we got a whole range of different environments which
means that for every single show we can have a subset of the tools that we use with a specific version where different the virtual might using completely different ones obviously the mind users themselves the artist himself something that's quick and easy to use something is consistent
don't want worry when using this is the axon would have used this way of doing it by using that at the end of the year that was we do not it mind the scientists are not necessary train developers but they do right told they had around quite a bit and we need to make sure that we can't presented data in the same way for us in the same way for them to want to expose only certain parts of a data to the minimizer this is the way and so we have develop this we have developers of any level coming some of them are trained in more of these effects other things also trained in as management but did not necessarily trained in anything that's distributed all you know spoken across the different sites around the world so to do this we developed a service-based architecture for them and we provide as a platform
as a service to all the different artists and developers and is the most probable set up with mobile transposable guarantees all be going into every single bit throughout the difference lies in this this is my introduction to what it is that we try to lighten the persistence right service for developers of any level to anybody that comes in and offers they should be able to write a Service during that they get into production by the end of the day so currently running our 2nd generation delivered in 2012 and has a lot of 2013 replace suffers generation which was push model which cause a lot of problems this is a request for comment would she starts feeling
with extra fresh running and running and running and there's no way for us actually then any sort of what so animals and you based model which allows us to things that I sent it to make sure that we have a specific flow can control the flow away when I swear so this is fast and
that's a lot of companies letters of before the lecture so have run the following the thousands of years but if you would have rendered which means no creating final image of 1 single machine would have taken 44 years that render which has a certain amount of time and about 65 650 terabytes of web data that was used as a result and the generated during the peak times but 50 thousand and then the request a minute which is 120 million requests and 8 hours
after those of you aggressive in Germany most of you have seen the Brazil during the game is about 4 times the amount of tweets that were about that specific that Congress Germany winning so another
distance you entered in setting up the whole system from the ground up I'm going to be starting with the actual service and what we have done
that is that service is nothing but a class so that make Europe of making this was about 20 minutes to make a move which is probability that sure but this right away so the of the direct because the doesn't work and which is typical hello world scenario anyone with is that is a class that's absolutely alone but was complete possible and you don't depend on any
of the these tools for any of the scaling of the manner in which is very important for us because we don't want to have to worry about any of these things and we have these little vector is equal to the public
was an appendectomy a private which allows actually expose what methods are available throughout the system for other people to use the public would mean that the artist and developed and users from outside and after that it would mean that you can call it from a different service so those who web service now but I should be doing anything useful and is the thing about help was going on and the lower bound of ratings that would have been tomatoes so there's actually do something this we provide the schools would call them and it actually lost all different services and when we do this by declaring dependency inside the class I can say I have a dependency on the storage service and here I'm using the source of the text and check if the data is on this guy can do to itself self-storage and techniques in the past and the parameter and at that point of course
Wilson using information about database itself over much of emotion and we can do that with what we
call infrastructures and infrastructure is really of formalizing this for us to formalize our access to the back systems databases loading considerations sessions and you those things and then you see the underscore that is the nature of the structure and just provide provide users with a nice clean and this is the way that access databases they are in themselves to but the
stateful services so that we can do things like pulling and and passing and those those things and they are local to the service of the service selection of spread across the system they are inside the same Python model which allows the pool of course and really realised that about this and then I'll be happening on this quite bit throughout the whole book is that we can swap any of those services for all the services so means that for example in this case the at the bottom you constant were
going from another consideration and in the development environment this could be a dictionary interaction is the maximal all beyond what floppy whatever floor and we because what that and out different services without once again this the actual developer of those having to change anything in this so that's
something that we know something about you know is not very useful in some ways it's not scaling is the local among persons machine and animals don't have that that 1 can actually provide a consistent interface but we did trade all the other structures that we need to contain and of possible already have with all of us that we might want to use in the future let me
introduce you to service provider and the society create 1 of the service
providers and this actually allows us to get that consistent interface and services for us and at the bottom you can see you know we try to make use of this year and source service passive and elegant and Paul with services that may serve as an alternate member United all logging and actually change lot this kind of things that we actually allow them to do but we still don't but still not the skill and use of the word and again
with the idea of proxies and proxies of standard
services for the requested service so that prevent the other that you want but not reasons that you want and underneath the hood to just take few data into you and that you can pass along the whatever I will also then the police use
transports and but once again the completely transparent to the user the user doesn't have tectonic service developers have to care where this data is coming from so
useful transports and they us extract way technologies like prevalent user and you PDB any of those you can
also track them away and it allows us to transfer the sort of things like about so that so there will use the reference to you in the next 1 is by and I think without once again and it was so was a rather anything else just after passenger transport that which configuration so at this point
we consider scalable but still going be expensive to run 26 thousand requests simultaneously with all of the services running up so Of course you images some parallel processing engines corporate and things the service developers us
shouldn't have to worry about how to do things concurrency and how that works well and they should had not been no it's something is that the CPU-intensive or I O intensive there is something that we do want to think about how we don't want that thing to think about long enough to include that there is that way that way that way we think that we should accommodate both the some the CPU-intensive or test and the I O intensive and we don't want them to worry about that 1 be able to you know you strategy 1 way introduces another way almost process if we want to so so far we have been building this this
little book here which referencing and what we did is actually stick a workable from the no look a whole provides a simple interface across various competencies and the fullest that progressed from internal use that is still you know by assuming from from argues in there once again workers can be changed can be
extended and it can actually change just like with with middleware thinking just below the whole nice so here and at this point know
that a nice little building block that we can reuse everywhere so this
point we really have something like that of building blocks that we need to start building a slightly larger system at this point and the nice thing about it is that it
makes starting blocks together and that's what we did in production so in practice we have a
cross-language pipeline as and you know we don't have Python with a 95 % of Python is that it is the most of tools of closing the Senseval most anything that is really really heavy on on on that this is reality of that we might use the post graphics we have some
jobs laying out some of the Web tools that we have a new level with little ones and we actually be able to represent all the data that a man has all services have to all of these different so that all of these different languages and I is the way so what we did is actually the office worker we
replace it might respond faster than nicely nice with and simple just a little so soon is it to see where changed and allows that to use HTTP quite effectively allows simple clients of every single language learning and language they should be able to make these people and it's a nice simple blind people can use of people that were looking at the
spreading to you distrust for investors what we take care of that and take that away and at those levels to native title the size the transport by the transportation this notation X Martin which available across those languages all but those limits us to native title only need to start extending the encoders and because section starting with those issues so from and here's my room is a fast worker and actually don't really do any work in fast except for session handling which itself an actual service the rest is just being proxied across the rather and around you discounted distribution across all the different so that we might have running so at this point we
got a system that can be distributed that is available to all the different languages of course when make sure fault-tolerant as well so what
we did is actually running 2 instances of might riskier fossil organisms that candidates in front of that to load balancing and failover nice and easy and actually among cluster red and set up the road and actually cluster those were from reverent you run multiple instances of around you what that gives us is that we can actually the use of proxies to consume from multiple cues and transport at the same time so that vote this fall any of these transport with different transport and then go as far as learning rather than q and other relevant you cubicles from relevant user and you and registered the same time can start consuming with 1 single proxy from all of these different transport at the same time so that if you to something nice comes along something better comes along it also changes we don't have to rewrite the services we don't have to rewrite anything else biggest faulty that's in and out so at that point that going the last that left is
military which I'm quite keen on and you know there's something needs to be done so what we did is we assign an actual Aidid every single request as soon as it comes in we should have an idea the idea is being followed systems that are go from service status of features service you Vancouver Isabeau's of event will have a trace of local bank because every single request is lower and I can actually start searching on those requests used throughout the system and find tradables and all the requests
so so it is very long services and you know so space architectures which may show that most of the sixers analog instance and so the data that we have in
here for example at the bottom of world have a calculator however long it takes to get from from that and so from Michael was the to the end of Ramadan Q of all the the amount of time it takes to execute the request we can map these
onto the system itself and which assembles the statistics service or looking service
and what that allows us to do once again is that if at some point we now using provenance somewhere when you start the within change statistics service without once you have it contains everything else the nice thing that we do with our workers that they can be wrapped in a new little of things with is that we have 1 single word to execute the request and that workers wrapped in statistics workers so as soon as requested being done handling and the reference was accused already have the result going back to the client that is the 1 where to start doing stuff calculation so we don't have the overhead of actually doing status while still executing the request is done afterward of course is a bit of an accomplished from all that it all happens afterward same thing for loading a lot room for a logging service which allows that to dynamically change loading levels of vitamin request that is a j Slovenia would be book for this specific service and we can make changes on the fly as we need them so
maybe it was wise we use all of those almost all take it out it's it's really cool and similar the publisher for those who know those invited and actually we extend it with the model so sold in our members of convertible framework for us and interact so clients over this so we actually useful investigate the system on the fly actually request to know what's going on in the system without really having to log into the muscle and was really gives a list of predefined repeatable configuration that we need about its size to look after I want you to make sure there was running inside a on the same society a society when
we should have all the same so we got an adaptable extendable configurable system at this
point we can change this is spokesman out like want to guess what transpose with whatever the tools that we need but with a big thank you to everybody who has been writing a lot of these models like the problem you are all you know simple Jason or whatever you use them
a lot thank you for that but it's very yeah extendable and configurable and it's all configuration based we can extract the whole system from you know system level all the way down service level and we have the best of breed system at the point where you know we might build a system but for particular sure pipeline all when you want to use so there's a couple things that was still looking at contain resistors
1 we don't want to say that you know this if you're going crazy on it to actually take on servers peaceful will again containerization looking at all the scaling is all but if you have an investigation just like us you want have a chat about it would be great but I was looking at the possibility of that open sourcing the whole system this
produces little technical things sorry I Dingell intellectual Python code itself and its 20 minutes so actually taking into in the required tricky this is a couple of slides we
all actually looking for people who got a lot of things in production at the moment of the jungle who wanted them out for that wants to be really only and of course we are
hiring as all across all use across everything so I have a website look at recruitment and open it up to me after the talk of course all
thank you and you have any questions
but microphone for consensus over there just enough to go there aggression the
and do deal with versioning of these services in any way we do with burgeoning of the services yeah services yes every every single services deployed as an inversion of and we haven't actually that's what we use all the tools that we have a configurations set of those services and this change we actually run up different versions of that and we can actually have a staging and development naturally can put those changes out 1st and run bunch of test against and spread out in a couple of users to start using before actually reduction to every single services is therefore for yeah I have a question how How among that differs from let's say the standard Enterprise Service Bus because I don't get it to I don't I don't understand why have you wrote the code from scratch and not used for example let's say a service vastly block you can plug in different services and so you mentioned something about that I'm I mean I'm I'm saying that he has been and the price of these but because the the when you do want to do that's say a service oriented architecture just use and usability and I don't know why I haven't you done that's that I don't but I'm not too familiar with with is if this is a technology tool like here and the price of his bosses who when you want to integrate a lot of uh a lot of hook and a lot of different forest environments and so on you just use the and the with 1 2 protocols and so on and this is looks quite saying to me and then I don't maybe we can travel that to get to those of interest to learn about cold political comical questionable call those and what do you say have to do it with a new organism and metrics sorry it has an area with a multiple of those things i want to hold you to use to do and localizing if the approach to be with for or something is so the front of the engine to use the load balancing about multiple might risk last instances of set up their own educational balance between that and actually used on the other side views reverent you that you load-balancing so we have of proteins have often been that have a certain amount of they can handle simultaneously and if we see that the user getting too long to start spinning up more services that's what we're looking at of scaling do with those issues they make and what you do about C March that about you know you have a service which operates on data like source images and and then also enable new laws obligation run world how you make sure that the data is available and have a good person not around the world so we we get the various things that we do we got 1 of our infrastructure that we have is what we call across much of the structure and of course we cannot check if something is not only so restorative we might have a sports event to the Commissioner of Oracle across national you can do with itself cross match with this site and you can then use the same service interface if you going and call the specific methods see in Vancouver to and check with the story of his event with available and the mind and of course sinking Q which securities sinking all of the data from all the different sites which happens at really start we have specific rules set up as part of a service that says OK this this method of this acid has been released doesn't need to be single any or all of the other sites so this'll service says that and the same goes for generating data on prevents generating like terabytes of data on but some officers to reduces to monitoring and you look at the operations and so so the request goes through a number of very small very lightweight so wouldn't be sending terabytes of data for the time we just use our sync async services quality and actually detect and make sure that the data that needs to be seen is going to be saying that we have a large dependency trees in these assets focusing on this as has this section this section the section you know it is kind of rates and going check if we need them in your sights is all it is doing something like lighting where it is the Renaissance but that and more questions don't that's averments made
Web Services
Distributionstheorie
Code
Computeranimation
Videospiel
Software
Web Services
Einheit <Mathematik>
Vorlesung/Konferenz
Systemplattform
Computeranimation
Programmfehler
Vorlesung/Konferenz
Softwareentwickler
Visuelles System
Chatbot
Computeranimation
Integral
Soundverarbeitung
Metropolitan area network
Bit
Desintegration <Mathematik>
Soundverarbeitung
Vorlesung/Konferenz
Programmierumgebung
Visuelles System
Bildgebendes Verfahren
Computeranimation
Gruppenoperation
Client
Computeranimation
Bit
Prozess <Physik>
Physikalisches System
Datenfluss
Computeranimation
Metadaten
Metropolitan area network
Informationsmodellierung
Arithmetische Folge
Vorlesung/Konferenz
Computerarchitektur
Bildschirmsymbol
Verzeichnisdienst
Gammafunktion
Schätzwert
Web Site
Subtraktion
Softwareentwickler
Atomarität <Informatik>
Datenmanagement
Quellcode
Programmierumgebung
Computeranimation
Übergang
Metropolitan area network
Vorlesung/Konferenz
Ordnung <Mathematik>
Router
Softwareentwickler
Subtraktion
Datenhaltung
Versionsverwaltung
Datenmanagement
Kartesische Koordinaten
Programmierumgebung
Computeranimation
Eins
Teilmenge
Spannweite <Stochastik>
Vorlesung/Konferenz
URL
Speicher <Informatik>
Programmierumgebung
Web Services
Soundverarbeitung
Softwareentwickler
Subtraktion
Web Site
Bit
Multiplikation
Extremwert
Wellenpaket
Übergang
Biprodukt
Systemplattform
Computeranimation
Übergang
Modallogik
Informationsmodellierung
Generator <Informatik>
Web Services
Datenmanagement
Rechter Winkel
Mereologie
Vorlesung/Konferenz
Computerarchitektur
Softwareentwickler
Resultante
Extrempunkt
Datenfluss
Quick-Sort
Computeranimation
Einfache Genauigkeit
Virtuelle Maschine
Metropolitan area network
Informationsmodellierung
Benutzerbeteiligung
Volumenvisualisierung
Vorlesung/Konferenz
Bildgebendes Verfahren
Web Services
Umwandlungsenthalpie
Metropolitan area network
Web Services
Twitter <Softwareplattform>
Spieltheorie
Proxy Server
Vorlesung/Konferenz
Physikalisches System
Abstand
Computeranimation
Web Services
Metropolitan area network
Zentrische Streckung
Web Services
Klasse <Mathematik>
Vorlesung/Konferenz
Vektorraum
Computeranimation
Logik höherer Stufe
Inklusion <Mathematik>
Web Services
Punkt
Datenhaltung
Klasse <Mathematik>
Physikalisches System
Quellcode
Bitrate
Computeranimation
Web Services
Vorlesung/Konferenz
Information
Speicher <Informatik>
Hilfesystem
Logik höherer Stufe
Metropolitan area network
Informationsmodellierung
Bit
Web Services
Trennschärfe <Statistik>
Natürliche Zahl
Datenhaltung
Minimum
Systemaufruf
Vorlesung/Konferenz
Physikalisches System
Datenstruktur
Computeranimation
Inklusion <Mathematik>
Extrempunkt
Logarithmus
Interaktives Fernsehen
Ausgleichsrechnung
Computeranimation
Virtuelle Maschine
Metropolitan area network
Web Services
Datenerfassung
Softwareentwickler
Datenstruktur
Programmierumgebung
Schnittstelle
Logik höherer Stufe
Web Services
Metropolitan area network
Web Services
Logarithmus
Minimum
Äußere Algebra eines Moduls
Wort <Informatik>
Quellcode
Widerspruchsfreiheit
Service provider
Computeranimation
Schnittstelle
Metropolitan area network
Proxy Server
Web Services
Proxy Server
Vorlesung/Konferenz
Softwareentwickler
Ausgleichsrechnung
Computeranimation
Standardabweichung
Punkt
Vorlesung/Konferenz
Konfigurationsraum
Transportproblem
Quick-Sort
Computeranimation
Portscanner
Softwaretest
Prozess <Physik>
Web Services
Datenparallelität
Zählen
Strategisches Spiel
Vorlesung/Konferenz
Softwareentwickler
Parallele Schnittstelle
Parallele Schnittstelle
Bildgebendes Verfahren
Computeranimation
Web Services
Metropolitan area network
Punkt
Proxy Server
Vorlesung/Konferenz
Extrempunkt
Computeranimation
Schnittstelle
Physikalisches System
Metropolitan area network
Task
Punkt
Proxy Server
Gebäude <Mathematik>
Physikalisches System
p-Block
p-Block
Biprodukt
Computeranimation
Benutzerbeteiligung
Web Services
Prozess <Informatik>
Formale Sprache
Vorlesung/Konferenz
Computeranimation
Office-Paket
Eins
Übergang
Metropolitan area network
Distributionstheorie
Punkt
Formale Sprache
Einfache Genauigkeit
Ausnahmebehandlung
Transportproblem
Computeranimation
Übergang
Zahlensystem
Client
Web Services
Inverser Limes
Vorlesung/Konferenz
Garbentheorie
Decodierung
Proxy Server
Multiplikation
Abstimmung <Frequenz>
Subtraktion
Punkt
Web Services
Selbst organisierendes System
Formale Sprache
Vorlesung/Konferenz
Physikalisches System
Computeranimation
Instantiierung
Lastteilung
Metropolitan area network
Web Services
Total <Mathematik>
Computerarchitektur
Physikalisches System
Extrempunkt
Analogieschluss
Raum-Zeit
Ereignishorizont
Computeranimation
Instantiierung
Metropolitan area network
Statistik
Web Services
Minimum
Physikalisches System
Rechnen
Computeranimation
Resultante
Umsetzung <Informatik>
Statistik
Bit
Punkt
Mathematisierung
Mailing-Liste
Maßerweiterung
Physikalisches System
Rechnen
Framework <Informatik>
Computeranimation
Übergang
Portscanner
Informationsmodellierung
Client
Web Services
Vorlesung/Konferenz
Wort <Informatik>
Overhead <Kommunikationstechnik>
Konfigurationsraum
Modul
Metropolitan area network
Informationsmodellierung
Punkt
Vorlesung/Konferenz
Konfigurator <Softwaresystem>
Computeranimation
Offene Menge
Metropolitan area network
Zentrische Streckung
Punkt
Web Services
Server
Vorlesung/Konferenz
Physikalisches System
Konfigurationsraum
Computeranimation
Übergang
Rechenschieber
Momentenproblem
Vorlesung/Konferenz
Biprodukt
Code
Metropolitan area network
Web Site
Vorlesung/Konferenz
E-Mail
Atomarität <Informatik>
Versionsverwaltung
Gesetz <Physik>
Synchronisierung
Computeranimation
Netzwerktopologie
Metropolitan area network
Dämpfung
Web Services
Hook <Programmierung>
Vorlesung/Konferenz
Schnittstelle
Softwaretest
Nichtlinearer Operator
Sichtenkonzept
Benutzerfreundlichkeit
Computersicherheit
Systemaufruf
Quellcode
Bitrate
Ereignishorizont
Benutzerschnittstellenverwaltungssystem
Garbentheorie
Programmierumgebung
Standardabweichung
Instantiierung
Web Site
Subtraktion
Selbst organisierendes System
Mathematisierung
Zahlenbereich
Enterprise Service Bus
Code
Multiplikation
Softwareentwickler
Datenstruktur
Konfigurationsraum
Bildgebendes Verfahren
Wald <Graphentheorie>
Protokoll <Datenverarbeitungssystem>
Linienelement
Matching <Graphentheorie>
Logarithmus
Anwendungsspezifischer Prozessor
Schlussregel
Ordnungsreduktion
Office-Paket
Summengleichung
Flächeninhalt
Last
Mereologie
Computerarchitektur

Metadaten

Formale Metadaten

Titel Amanda: A New Generation of Distributed Services Framework
Serientitel EuroPython 2014
Teil 70
Anzahl der Teile 120
Autor Eenbergen, Jozef Van
Lizenz CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/19987
Herausgeber EuroPython
Erscheinungsjahr 2014
Sprache Englisch
Produktionsort Berlin

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Jozef - Amanda: A New Generation of Distributed Services Framework To help create award winning visual effects, MPC developed a distributed service-oriented platform, Amanda. Amanda allows developers of any level to write a service that is presented to users across 8 facilities globally without them requiring any knowledge of building large concurrent systems. It allows artists and developers across different domains to work with clearly defined API's and gives the service developer control over what and how data can and should be accessed. The talk will cover how to set up such a platform from the ground up. Starting at the service level building it out with additional modules and technologies until the fully distributed system, covering topics such as concurrency, componetisation and monitoring that allow the fine tuning of setups depending on the type of work being undertaken and changing business needs.
Schlagwörter EuroPython Conference
EP 2014
EuroPython 2014

Ähnliche Filme

Loading...