Merken

The inner guts of Bitbucket

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Beta
Erkannte Entitäten
Sprachtranskript
ch
OK Eric is going to tell us all about BitBucket with a focus on Gaiseric this is just by OSHA just focus on getting it is was lot faces 1 and I think I've ever seen in 1 room staring at me from and the other goes by so America in time with the lesson and I work on on that but it's 1 of the more back
and the developers of the bucket and until you all about the
but it's architecture and infrastructure and or at least as much as I can in 13 minutes before that they
want to share with you this photo and those who don't easily recognize the rocket here this is a 7 5 rocket it's the the rocks from the Apollo program the CIA the moon work and that's the 1 that of that particular 1 that got out and from 2 D is the more than that In show it to you because it's like the whole Apollo program is I find a like a fascinating piece of history and sure enough along here but it's a bit like this rocket when they built this again against the the program around it so the pinnacle of innovation in engineering at the time In a goal that they set out to achieve this so ridiculously ambitious in the sixties and the amended we're bringing him back when the I guess state-of-the art was the Russians which is flying a chunk of metal orbit around was quite something enormous undertaking I think at some point like 500 thousand people were working on it ridiculously large billions of dollars a year but it works you know so you must have assumed that you know only really the the smartest people work that you were able to work to
pull this off so quite literally what time is not from the murder in the earlier this year I actually went the Florida in visited the Kennedy Space Center in taken every year and they've got 1 of these these things on permanent display here is here is that an actual remaining
Saturn 5 rocket that they've they've taken part into the the separate rocket stages here and you see that see of course you can see sort of what's inside in which struck me when I was there and I looked at this 1st on the scene is that is that it it looked so and simple awareness of the right word but rudimentary perhaps as in a very functional but look at this thing what it's like a sheet of rolled up metal around a I guess a massive yesterday I mean it's really not much more than mean of some planning but even that limited but in you know I guess I never really considered really considered will be inside a rocket like that to be able to do the things that that but so I guess I should
expect something more complex you know more ingenious of from it's a similar story appear at the back of bottom sort of you know it ends
here there's a there's a flat surface and then you will just engines on the at the bottom here are a few days actually see that the engine mounts like this using everything so really policy bolt protruding everyone yeah no I don't mean to disrespect the Apollo program by the way I mean it's it's still as amazing is that as I thought it was but it's sort of senior staff of closer and hour make it more approachable in it going down to earth if you will the anomaly anything that is not
representative of how we but tend to perceive technology we have a high regard we don't really know much about we tend to assume that things are more complicated than they really are here that the people working on it are by definition much smarter than we are you know the whole grass is greener things but in is that the potential perception that I wanna debunk today by laying out the architecture uh behind the bucket you can also at the same time share some anecdotes and I guess you know somebody instances were we we screwed up so if you a little bit like me that if you tend to assume that other people are smarter than you then you'll be glad to hear that usually no rocket science behind a bit bucket in you know everything and that is running now sort of built from the same tools that you will use yourself to sort of try to
break it down a little bit so this is roughly the architecture of the market Ivan I separated into 3 logical are areas through is the the web layer which is responsible for a load balancing high-availability that kind of stuff then there is the the application layer but that's where you know the bar code is that all the Python stuff is a big bucket is almost exclusively written in Python only in the last storage layer where we know we keep a repository and all that to talk
about its lair individually and time permitting all share some anecdotes so the the 1st layer of that layer and consists really of 2 machines only to the bucket is all there's no virtualization bucket we're on real hardware we manage them ourselves we have a data center in the US in that we have to global assumptions In those events they only the 2 IP addresses the ceiling you resolve the above other work in these machines basically run into next major proxies yeah but traffic it comes into 2 bulldozers 1st hits and genetics and genetics for those of know it is a it's an open source web servers and it's it's pretty good at SSL but it can also be used really well for the reverse proxy and as we do here in this layer and so when I request comes in and it is encrypted everything that Bob is always interpreted so the 1st thing we do is strip off the and the approach and that's done using engine next year and I wanted to be corrected we fought on Twitter proxy which runs in the same machine it's a proxy also an open source of reverse proxy server but really good at doing load-balancing and and fell over we have a whole bunch of that servers and so that it's a proxy inspects the other request invasive properties decides how to fought on all the labels for long to 1 over many actual application servers and all there is another engine next instance isn't so this engine also just a reverse proxy server is not the actual web server it takes care of things like that request logging compression response compression here in an asynchronous response in request buffering and and that's why logically it's part of the Web layer because it doesn't actually process requests and then ultimately that forwards it onto the real Python web server on the application server now so that States yes we also do SSH him as stages of Dixville different paths as the stages that is it's a different probably can't easily and D. kristofferson so we do need law so it does go through each approaching and just as a TCP connection here and and it's approaching affords it onto the least loaded back so that passes a lot simpler but make no mistake not necessarily easier to run that reliably from his we we find out really just recently when the user started to complain about SSH connections dropping out sometimes what users say that they get hung up on and looking at the error messages it seemed like that was indicative of a capacity problem but we had like not enough capacity the suicide basically to handle the request rate of monitoring tools full was a different story that we had plenty of capacity so we stand for a little while until we we started by analyzing the network traffic on the the yeah and in particular we looked at the the frequency of syntax that were
right and so since packets and as a part of the city and it marks the start of a new TCP connections and so time something those but each single 1 of them gives you really good accurate view of the incoming traffic you see that here so we can see here is an interval of 16 minutes over which we are captured every syntactic it and you can see right away that is ridiculously spiking you the spikes are aside from being very high everything in so pretty evenly spaced count them using the 16 bytes interval 60 minutes in an hour this no coincidence the spikes at the start of every minute like precisely at the start of in the last about 1 to 2 seconds only we can see the rate at that point is ridiculously like 3 or 4 times higher than our average load
working theory behind this is that this is the result of 2 thousand continues integration service all around the world that are configured to periodically hold a bit but we goes on here that in combination with the NTP everybody I guess these days using Debian clocks are really accurate here this yeah there was a bit of a problem because even though we we have enough capacity for the average rate of during the spikes we actually don't have enough capacity not solving this and it's not we got really come to war SSH infrastructure to be able to deal with the large by so we can instead is men back into the the web layer into it's a proxy work we basically have a hook into that traffic comes in here and we can figure is approximate to never forward traffic at a rate higher than what we knew our capacity dictates but then don't make any change the ingress and so during the spikes it's proximal happily accept all the incoming traffic other 1 actually connect for all the train connection at once so spreads it out over in a few seconds another year like this graph on the applications there was a lot smoother than it is on the on the wall society the probably see it if you have a con job unit started and that very started a minute every time 1st any other 2nd will probably have a few seconds left the line so it's a bit of a
funny for problem never really considered until I crept up the probably won't have a really with websites that have use click on links of but if you operate like a public API that is very popular in people scripts against that you might see some origins the application layer this
role the magic happens to have a reserve the website runs this this
layer is distributed across many tens of of service research in which the all run a whole bunch of stuff there on the website the website is a fairly standard Django at really like that but it started out as a pretty much hotter presenting weapon still very important the year and we want that I'm gonna go on the web server identical in Python web server that is relatively simple and we're run in perhaps the most basic configurations we use a single workers meaning that there are processes that process 1 requested time so we have a whole bunch of processes and multiprocessing to get concurrency but in the end SSH we handle SSH using really just a standard organization the server diamond are the same 1 that you will running a Linux machines and laptops in with 1 difference so we we made a small change with
a small patch that allows us to use the database to look at in a database for public key by open a stage the looks on the file system it's hard wired to file system defined properties that's not practical for us so we have a little change to make that happen and that it is the standard open Society sir so we don't need to maintain a I will send you on back on processing to any East Java process that we can't guarantee you respond in a few milliseconds we dispatch off to to a background system you and this comprised of a cluster of high available on the random and servers right use the is an open-source Erlang implementation of aim to be broken Robert here to consume jobs we used celery so we have a whole foreign salaries distributed across all the machines to process these jobs example there are a few for her report for instance there's actual copying the files involved so that might not complete immediately and that's this this dispatch so it looks very basic
at the end of the day it's it's the same components that you will run distributed state mostly across most services nothing really special about it a simple is usually good year and we have this set up for years and years and that but it is now think over 30 35 times bigger than it was when we started we acquired images in a and this held up really long however you can still scrub as we do from time to time in 1 of
those examples was on when we decided to upgrade our password unsolvable at time we have restored patterns in a database that we stored
sold shall on actions to very common in so that means that if somebody for some reason gets a whole of a database of they only have passed values and so on and they still have your password shall on hashes for passwords are so slowly being phased out and replaced by more strong secure hashing over yeah and the reason for that is that is that even can't decree shall 1 has to get past what we can do is think of a word that might be password compute shall 1 and then compared with what's in the database and if you just think of enough words to triumph combinations you might brute force password so we have a strong password chances of anybody brute-forcing that through show on our because some cryptographers maybe here in the the ring but let's call it negligible but however we have millions of users on the market and not everybody has a strong password in the password that is a word in the dictionary they're not going to tell you I guess that's why it's a whole different story is there really aren't many words in the dictionary certainly not when it comes to computer computing challenge values for it 0 yeah this short forcing people to not use simple passwords and another thing you can do is set of greater two-way stronger hat now what these things do nothing special really but the hash value or hash algorithms that are more expensive deliberately more expensive by rehashing their hash value over and over again thousands of times deliberately spending more CPU cycles in and that's all we want to operate let me show you just how big the differences so we want to upgrade to be critized the critters 1 of the more modern art iterations style cryptographic hash algorithms in and we compare it with shallow 1 2 descriptors measures how many hash values you can generate in 1 sector before B Chris can see that my laptop was able to this this uses Jangles code to generate hashing algorithms electrode
with the required required with the optional C extensions to make it as fast as can allow a laptop that is massive 3 hashes per 2nd for the correct 1st 160 thousand for salon so it's 5 orders of magnitude more expensive to CPU cycles as it is absolutely huge as great because it means if your weak password remains that a chance the has realize it as a server you have to incur the cost of that massively expensive calculation every single time somebody uses password for authentication we're really popular high-volume API in a lot of people use basic off for authentication so less itself it's it's not it's plaintext benefits that means that we have to compute the for every single request In API requests a relatively quick like on average of independent tens of milliseconds to can imagine if you added 300 ms password checked to every single 1 of these requests yet problem it we did is we naively roll this out in instantaneously to what's up and down whole course
old also the use of to a higher status in of the correct a realize the mistake fairly
quickly obviously not quickly enough that fairly quickly so we roll it back relative to downtime the minimum and then we had a problem because still have move away from shall 1 become really make because it's cheaper actually you can but you don't want to because the whole point is to have an extensive algorithms in so we could do however is do less of in was people use the API tool it and write a client it typically do more than 1 request in quick succession in on instead of using the same password over and over again and will be computing saying the creed over and over again and so we decided to implement a sort of two-stage hashing system where where request comes in the instead of computing the expensive the rates we now compute a old-fashioned sold shall on value and then we use it to look up in a in memory a map dictionary and the be criticized as that's empty in the beginning we then compute the be Grenville yourselves check it against database see the correct and then stored at mapping like shall 1 forces of the critters in that memory table and then the next request that you make the same password is able to look up to be good value from the in-memory cache that that we were able to cut out guessing 99 per cent of all the than the credit calculation and this it's important to understand the ramifications of this system right because you might be tempted to our to think that will be positive we can you work your be create authentication down will be back down to show on the strength of this whole story but that the important thing for the main thing I guess is that shall on values never hit cold storage endorsed databases will correct so if you get a hold of of the database still only have the crypts in even if you able to somehow tap into our series in and get a hold of a copy memory access then on you get some shall once you only get show ones from the users that are active at that very moment because these cash centuries that's essentially what they are parts expunged very very quickly so there's we were able to walk to get this thing
running in breakthrough it was to be created that aren't even than the they do remaining of 1 per cent of the time we spent on the grid is still very significant very significant just look at the ratio of a hundred 60 thousand 1st 3 yeah and right now today to look get 1 of our servers in you run like a 1st proper sample use the that the uh be criticized for method and is the the most expensive method that runs on the machine here at any point in time I think it is like a 12 per cent CPU so it's still hugely expensive up to any future against we should probably be looking at migrating were offering some like an iterative basic off
maybe you know standard relatively standard states people still consists of which are comparable here and have a a limited privilege that the analyzed multimedia the storage layer so here
we we keep track of the data obviously
the biggest amount of data the store the course of the year severe repositories the end of the there are millions of lines repositories here we decided to keep the storage of that as simple as we could like sort of in line with with everything else you've seen so far on it we decided to just ordered stuff of file systems I like to do a new new location time I get a mercurial where were designed for file systems work really well you can as a post for instance to modifying the material to be able to talk to you the like a distributed cloud-based objects system of some kind of principal coders we decided to keep it simple the other file systems the live on specialized appliances when there commercial companies in are accessible from the application servers simply using a fast it aside from that we we have other we have no sequel storage like distributed which match systems that readers can Memcached weirdest for your your news feed in the repository activity the DC but in the main will basically everything that is transient we can lose and then they the data
for the year mind the website is all stored in the traditionally justice equal so we use a post-processual in in the the data is manipulated and access basically exclusively through the general around here and it works worse pretty well using is that SQL databases posters nodes and exceptions are generally kind of hard to scale beyond distinguishing but transparently should so like unless you go implement applications charting to to to separate your data across multiple databases transparently scaling SQL database across multiple machines is entirely trivial so far we have not kept things simple we're running a single post press database but it's a very very big machine has no trouble the point in for for high availability we have several real-time replicated hot slaves stand by the future should that thing ever to become a long and then which hopefully will be that means service popular I guess we have
to look into it a shot the the a talk about this stuff all along the and and might do so we there's only 30
minutes you only a few minutes left at this point so I wanna leave this if you have any questions and I'm happy to take some now where we don't know what kind of text now otherwise check to me afterward so we also have goes in there lower level so you can just find is there in an otherwise
we have like to invite you for a drink tonight we're hosting a drink up in a 1 year bond staring at 7 my house and saying yeah
so you like to invite you over the coming drink as you can all of this stuff to my colleagues to use rows of firing solution a book that is also possible that was thank
you very much for this and you hope to see also that the few
thank you would like to speak about the income instances sons of if you want to take any questions that you but any questions the question about the fact that the G protein and engine exists in the UK from the beginning of the request so as you proxy actually has the so support we tried that but yeah that's of so the I'm like our set up on web queries a little long convoluted many but there are a lot of components is shown here in the strict that's not strictly necessary part of that history of organic growth in the historical so we it's a proxy has always been very good at SSL at least not in our experience in that we've experiments with a kind of different terminators use stumbled a bunch of others in the in at some point we found that engine X was at least for us the most reliable and so we've left it there and I noticed situation is therefore change in it's approaching so it is something that we intend to read revisit some point in the future this can you get a so I noticed that the bucket uses quite a lot of JavaScript on the web server side and I wonder if you use WebSockets and if you do what do you for them on the server side all right so do we use WebSockets no we don't currently use electrodes we've experimented with WebSockets quite a bit of for things like with real-time notification and pull requests for into those kind of things but that no 1 actually using thank you also give you the but the bosses were used for the the answer is that you might see and so on I am I feel about bounces but I was looking at the time to to go into it it is not enough time to go into all that right now we there so I invite you to to check later on but did you bounce a is a is a that post breast connection pooling they may be and so we we use Django Django but full doesn't come with any connection pooling and so on getting general to talk to a database to efficiently is is a bit charges on a challenging you need something else here and us a PG bouncer is a is part of the posters projects here and basically what it does is it makes stateful connections along with connections to the database limited amount and then you can figure out general to talk directly to the bouncer it acts like a database the and so then you Django open and close connections at a very high rate because the serving lot connections and there is a lot cheaper than opening closing actual database connections and so it bridges between the 2 come to us to make it more efficient and to also be able to limit the total number of connections that you end up having on your database is a lot more to it that way so we actually if you notice we have 2 layers of PG bouncin isn't a good reason for that but yeah so let's talk about that afterwards because there's no time called thanks to the north hi thank you talked about too much you know the the companies like of the large missions you as far as I could understand that physical motion so what happens is that because there is a the the physical machine goes down we have we have several of real-time replicated hot slaves to string replication were post-processed to have to have a bunch of slices and she goes down entirely then we take the we still the IP basically being in you know almost instantly hopefully move over to the other this never happened but but yes the figure that way the missile that
Code
Fokalpunkt
Große Vereinheitlichung
Innerer Punkt
Computeranimation
Informationssystem
Bit
Punkt
Digitale Photographie
Orbit <Mathematik>
Computerarchitektur
Optimierung
Softwareentwickler
Große Vereinheitlichung
Innerer Punkt
Computeranimation
Arithmetisches Mittel
Trennungsaxiom
Demoszene <Programmierung>
Datensichtgerät
Mereologie
Besprechung/Interview
Automatische Handlungsplanung
Permanente
Kontextbezogenes System
Quick-Sort
Raum-Zeit
Computeranimation
Flächentheorie
Stab
Minimum
Optimierung
Quick-Sort
Computeranimation
Bit
Dokumentenserver
Gemeinsamer Speicher
Selbstrepräsentation
Hochverfügbarkeit
Speicher <Informatik>
Kartesische Koordinaten
Quick-Sort
Code
Computeranimation
W3C-Standard
Metropolitan area network
Benutzerbeteiligung
Flächeninhalt
Last
Mapping <Computergraphik>
Kontrollstruktur
GRASS <Programm>
Computerarchitektur
Speicher <Informatik>
Normalvektor
Instantiierung
Proxy Server
Server
Subtraktion
Punkt
Prozess <Physik>
Virtualisierung
Gemeinsamer Speicher
Kartesische Koordinaten
Zählen
Gesetz <Physik>
Netzadresse
Computeranimation
Rechenzentrum
Metropolitan area network
Virtuelle Maschine
Puffer <Netzplantechnik>
Last
Benutzerbeteiligung
Reverse Engineering
Reelle Zahl
Endogene Variable
Quellencodierung
Einfach zusammenhängender Raum
Sichtenkonzept
Hardware
Kategorie <Mathematik>
Open Source
Zwei
Kanalkapazität
Bitrate
Frequenz
Natürliche Sprache
Ereignishorizont
W3C-Standard
Twitter <Softwareplattform>
Last
Mereologie
Server
Fehlermeldung
Instantiierung
Aggregatzustand
Resultante
Einfach zusammenhängender Raum
Proxy Server
Server
Web Site
Bit
Wellenpaket
Graph
Zwei
Mathematisierung
Schaltnetz
Kanalkapazität
Kartesische Koordinaten
Binder <Informatik>
Bitrate
Physikalische Theorie
Computeranimation
IntServ
Benutzerbeteiligung
Einheit <Mathematik>
Prozess <Informatik>
Hook <Programmierung>
Skript <Programm>
Gerade
Virtuelle Maschine
Rhombus <Mathematik>
Benutzerbeteiligung
Subtraktion
Web Site
Web Services
Prozess <Physik>
Zehn
Selbst organisierendes System
Notebook-Computer
Nichtunterscheidbarkeit
Server
Konfigurationsraum
Computeranimation
Public-Key-Kryptosystem
Prozess <Physik>
Applet
Mathematisierung
Implementierung
Computeranimation
Virtuelle Maschine
Web Services
Prozess <Informatik>
Dateiverwaltung
Zusammenhängender Graph
Bildgebendes Verfahren
Kategorie <Mathematik>
Open Source
Datenhaltung
Hochverfügbarkeit
Physikalisches System
Elektronische Publikation
Patch <Software>
Offene Menge
Server
Erlang-Verteilung
Verkehrsinformation
Instantiierung
Standardabweichung
Aggregatzustand
Subtraktion
Datenhaltung
Schaltnetz
Gruppenoperation
Iteration
Computer
Computerunterstütztes Verfahren
Zentraleinheit
Auflösungsvermögen
Code
Computeranimation
Unterring
Algorithmus
Forcing
Kryptologie
Notebook-Computer
Mustersprache
Hash-Algorithmus
Dreiecksfreier Graph
Wort <Informatik>
Passwort
Zehn
Stochastische Abhängigkeit
Einfache Genauigkeit
Rechnen
Bildschirmfenster
Computeranimation
Konfiguration <Informatik>
Metropolitan area network
Mittelwert
Notebook-Computer
Dreiecksfreier Graph
Zählen
Server
Authentifikation
Passwort
Passwort
Größenordnung
Maßerweiterung
Punkt
Momentenproblem
Extrempunkt
Zentraleinheit
Computeranimation
Eins
Virtuelle Maschine
Client
Kryptologie
Stichprobenumfang
Passwort
Passwort
Speicher <Informatik>
Datenhaltung
Physikalisches System
Bitrate
Rechnen
Data Dictionary
Mapping <Computergraphik>
Forcing
Rechter Winkel
Festspeicher
Mereologie
Verzweigungspunkt
Server
Authentifikation
Tabelle <Informatik>
Multimedia
Dokumentenserver
Speicher <Informatik>
Fortsetzung <Mathematik>
Kartesische Koordinaten
Physikalisches System
Cloud Computing
Quick-Sort
Computeranimation
Weg <Topologie>
Hauptideal
Objektorientierte Programmiersprache
Ablöseblase
Server
Dateiverwaltung
URL
Speicher <Informatik>
Gerade
Instantiierung
Aggregatzustand
Standardabweichung
Web Site
Punkt
Datenhaltung
Hochverfügbarkeit
Speicher <Informatik>
Einfache Genauigkeit
Kartesische Koordinaten
Ausnahmebehandlung
Computeranimation
Virtuelle Maschine
Knotenmenge
Multiplikation
Web Services
Differenzkern
Lokales Netz
Punkt
Hausdorff-Raum
Computeranimation
Übergang
Proxy Server
Subtraktion
Bit
Punkt
Total <Mathematik>
Selbst organisierendes System
Program Slicing
Zahlenbereich
Abgeschlossene Menge
Hausdorff-Raum
Bridge <Kommunikationstechnik>
Computeranimation
Virtuelle Maschine
Datensatz
Benutzerbeteiligung
Datenreplikation
Radikal <Mathematik>
Zusammenhängender Graph
Figurierte Zahl
Einfach zusammenhängender Raum
Datenhaltung
Abfrage
Bitrate
Echtzeitsystem
Benutzerschnittstellenverwaltungssystem
Offene Menge
Mereologie
Server
Projektive Ebene
Aggregatzustand
Instantiierung

Metadaten

Formale Metadaten

Titel The inner guts of Bitbucket
Serientitel EuroPython 2014
Teil 99
Anzahl der Teile 120
Autor Zijst, Erik van
Lizenz CC-Namensnennung 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen.
DOI 10.5446/19968
Herausgeber EuroPython
Erscheinungsjahr 2014
Sprache Englisch
Produktionsort Berlin

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Erik van Zijst - The inner guts of Bitbucket Today Bitbucket is more than 30 times bigger than at the time of acquisition almost 4 years ago and serves repositories to over a million developers. This talk lays out its current architecture in great detail, from Gunicorn and Django to Celery and HA-Proxy to NFS. ----- This talk is about Bitbucket's architecture. Leaving no stone unturned, I'll be covering the entire infrastructure. Every component, from web servers to message brokers and load balancing to managing hundreds of terabytes of data. Since its inception in 2008, Bitbucket has grown from a standard, modest Django app into a large, complex stack that while still based around Django, has expanded into many more components. Today Bitbucket is more than 30 times bigger than at the time of acquisition almost 4 years ago and serves Git and Mercurial repos to over a million users and growing faster now than ever before. Our current architecture and infrastructure was shaped by rapid growth and has resulted in a large, mostly horizontally scalable system. What has not changed is that it's still nearly all Python based and could serve as inspiration or validation for other community members responsible for rapidly scaling their apps. This talk will layout the entire architecture and motivate our technology choices. From our Gunicorn to Celery and HA-Proxy to NFS.
Schlagwörter EuroPython Conference
EP 2014
EuroPython 2014

Ähnliche Filme

Loading...