Add to Watchlist

Using python, LXC and linux to create a mass VM hosting, managed by django and angularjs

58 views

Citation of segment
Embed Code
Purchasing a DVD Cite video

Formal Metadata

Title Using python, LXC and linux to create a mass VM hosting, managed by django and angularjs
Alternative Title Part 1: Backend and Architecture
Part 2: Frontend
Title of Series EuroPython 2014
Part Number 8
Number of Parts 120
Author Kraft, Daniel
Roch, Oliver
License CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
DOI 10.5446/20044
Publisher EuroPython
Release Date 2014
Language English
Production Place Berlin

Content Metadata

Subject Area Computer Science
Abstract Daniel Kraft/Oliver Roch - Using python, LXC and linux to create a mass VM hosting, managed by django and angularjs How we created a scalable mass VM hosting for open source web apps with python, LXC and linux with a web-UI based on django and angularjs. We'll show the underlying architecture of this service, several linux internals that make this possible and we'll talk about bitter failure during development. This talk will be python- and linux-centric with some hints for integrating angularjs into django.
Keywords EuroPython Conference
EP 2014
EuroPython 2014
Series
Annotations
Transcript
Loading...
hi everyone thanks for being here I'll talk about using Python Alex C. which is Linux Containers and of the notes to create a mass hosting managed by a Django and angular jails we have a schedule part 1 is my
part think and an architecture and how to is on his part as the front end in genuine angular
to no 1st part based an architecture 1st all me have done it cost from the 90 and my twitter account is
1 done dumb I am doing computer since 1984 1985 sorry and I online since
1987 obviously not the internet at that time who are we we are creating a for preconfigured ready-to-run
virtual services with roots for many open source whether think of it like the 1 taken so
all this hosted in Germany by the way with 100 % renewable energy so and that's how it looks like you
basically choose some templates like genuine that example to the version of it
and given the name Click on content is you only now
you have as contained in red because of its to Adolf then turn it on and then it becomes
greed then you click on the URL on the reachable at appear and there is a genuine and
is how it works we have 2 parts
in this architecture the bank and is called con that short for content management in the front end is called site is just our name can't has 2 modes 1st load is it could be around of the it it can be run as a daemon then it is it's a lot XML-RPC otherwise
it's an XML RPC client to its own demons so you basically you started as a
service to 1st and then you can also use it as like a console script which connects to its own XML-RPC server so this is how it looks like if you
don't call this demon shell
script with I don't know if you can read that that's not so important it's it's more about these are this is anything you need to manage
virtual machines on host you can but I can't here after that you can build Halesowen so example shortly you can remove templates creates containers duplicate content start and stop them and so on so can't call sales so it
needs its own dog food because it calls its own x amount methods just as likely to yield the shortly as the cytoplasm it can be
called balanced like the sides it contains anything needed to where was eventually many machine so a very important part is called works completely without sites that means we can use the server part tested individually only that we can granted locally although user management or something this is just a virtualization the and layer the site on on the other hand based on genuine and images calls column which
can be many the XML-RPC it does accounting and payment it creates the invoices it manages user accounts and
the registration and so on and also the site works without Congress also important to test it and to run locally and of course 1 can't doesn't run or isn't available you won't see any containers now that to we
re-implemented an existing solution for repeatable bills it looks like
that maybe someone knows what that is the top performers were less than so we essentially do the
same this is because of the history of column we 1st ran on topic but doctors didn't have the features we require so we added the features and had about 80 % code on top of talker and just 20 % managing partner and at some point we threw away the top priority and 3 implemented the 20 per cent of cells and this is part of it so this is a very very
simple language that essentially starts out they Alex
is runs the command inside like that it update and close that container again and for the next command it creates a new snapshot
of this template that's now was configured with command and around for example it operates that's why this results in a
large family tree because
what you see here is 1 line is a snapshot of 1 commands you saw earlier so 1 1 file that we actually
called con file 1 of 1 line in this final results in 1 line here in the hierarchy and the longer lines of final templates that you can you into so using Alexi volatilization shown in the books for the Web Council I potatoes for network accounting all the nodes tools
you of some rules for that if you you know but and it's
using a lot of C group magic from the Linux kernel for accounting like these if you act groups where the few lectures such counts the nanoseconds per 2nd
which I used on this if you and the same for the memory which can give ourselves active
inactive memory firemen caches and so on it's using also for storage is related files system you can mount
and any number of nearly any number of yeah ordinary directories on top of another entity derived from each other that means if you have a file in the lower containing file the content so in the in the law directly and offer the entire directory and demand both then you see finally and and they do some magic with deleted files and so on this is a very stable solution so that leads me to the failures we have many let's talk about the
3 of us we 1st choose the tree s instead of all
because it's fast it's even faster for millions of files that works the 1st it works the it has writable snapshot that means you can at any point use any sample volume in a B-tree FS make a snapshot of it and write in
both and that emerge it is
life quote time with some volumes that means you have at any point in the disk usage of this volume in some with all the snapshot lower and just to do what is the difference to where it was much of it from and it has instant and creation of selection it's like 10th of a 2nd but uh maybe you have
seen that this is the I will stick with diagram of it as you can basically anything in the notes on top of another and
like like block devices and file system than the furthest in image then partition inside it and and and so on without knowing exactly that it's not needed we use Device Manager for rate this is on the hot rests the physical outlets on top we use 11 PM the logical volume manager top we used to I 0 this is the case the virtual disk layer on top used to partition the on top of that the 1 partition we use text form on top of that we use an image file which we mounted as loopback device and it would be treated as this was a test set up because the image of this quite nice for handling and pick up you can just turn off copy the image files and right again however then the beach if unit that be driven is a lazy files estimate it does what needs to and cleans up later that means the bit-reversed has to run and it has to clean up later and died during its just every last data that's not meant to be be the best thing in notes In the next I'll
stick that's called barriers this copied from an article on television and in a sense that the
barrier properties the writing of any blocks after the barrier until all blocks written before the area are committed to the media that makes sure the joint of the file system is consistent looks like the buried find its way through these there's so some point in the state obviously after dividing it did work with berries so we try again the same basic set we use the rate and used still the amount of we use that I with ABM because it's already false set up
we didn't want to throw that away it helps us very much with the cups and things like that so
on top of that we used to partition and directly at the tree of test and you raise we tried to Crested again it didn't so it can be treated as look stable from that point barrier well that's another thing about to be traversed clear it
produces a lot of memory fragmentation if you have never heard of memory fragmentation years exists and Linux as a
table of that you will see when you you see a kernel prospective demons and 1 line of that is the spatial location failure in that case order for the order is the potential of 2 of the block size and memory that could be allocated this means that a 64 kilobyte blocked wasn't available of continuous memory this is
pretty good because there's not much and there's no fragmentation tool in
the Linux kernel if you have this state it will never run again except those memory for you OK so well we through the tree is presently and use whole which is a bit slower but much more stable and we are happy with it next data item on the
1st assume of you really excellent tool and it's pretty fast it has a good the serialization use can
basically just fire off messages they will arrive somewhere and it's a lot faster than XML-RPC but it was leaking fighters could do when not using G events with which we can because you are currently bound to the threats and then we use XML-RPC but it was very nicely with little a little bit and that to i'd Germany we used bytes for anything that was transferred like 4 memory
usage for network traffic for disk space and
so on but there a true the potential to the to the 31 limits of XML-RPC and we could use bytes anymore so we had to utilize all large numbers to strings moved to megabytes with repulsive so it's running for now until we therefore for the government limits again and have proper megabytes to what that is for my part I would have a lot more space but 2 times running and I'll give over to 1 of the 4 questions I'll the there directly after his part the but I and of the work in the front
end of 2 0 man and the the
bias choose genuine languages to get that you on the spot about 5 had the understanding a content was management and as a to a problem that has not they item adjacent to the eye and a bunch of other
individuals 1st of all those generous using this user protection lot of used fuel accurate and well so we have to telling you just to take the torque from the cookie and send it
to every asynchronous requests
then next problem languages standards with lot genuine genuine template language because they all use the development process so we have tended to use for example can
comparison policy In the nationalization J. uses the overflights and pretty much
the like so old no 1 that's the 1 to have a consistent state between the chain views and the jobless profuse you can use the chain if you society and job that which takes the
fights and that generates the opposite you can include into side and you function like a text to have that sensation we will use the very frightening images so for
example and the next day and we have a lot of requests depending on the was also commission so we have to include Commission United that is delivered by channel but
standard at the service languages and and the whole so you have to have greater customer it is set the for that so we have actually permission and so so you
can handle a request because response in response around the case is responsible for all 3 so we so that the patient was that that's the front page and the
registration of page a lot of the correct is that the entry bookkeeping stronger competences you full
trail you have always that and and all present in 2003 so you won't lose any money simple you
have to define the source so bank account of this nation account the user the amount and the user and that is privileged to transfer the money the most important thing is that they're keeping voltage on the front page of 2 you have a list of your content of
running that is updated every 2 or 3 seconds if you miss a trick by an idea In addition to
replace the old on every 2 or 3 seconds so you won't be able to interact with all your containers because just click although most governments replaced by another 1 the yeah it's all pretty much simpler but it's due to the fact that China and India are since program and the thank you Daniel thank you all about any questions come to microphones and our last winter so let's start with this chapter so what kind of version of the Treaty of 1st used for tests or what version the views for each reference that we're starting with what the hell of 1 to 30 in 10 and were testing again on 14 or I think it's 1 point something in in the latest version whether the travesti also that these memory fragmentation things and my during the whole process so it's is in to chase some of from and why is this marketing but and it was a decisions for development speed the XML-RPC module in Python is well tested and really complete and you all you have to do is um and know do I think from the XML-RPC class and it makes the server automatically you you you have no as you justify methods that can be called from outside this is just for development simplicity so you switch away from doctors of some features were missing doctors extending current development and do you think they will catch up with the features in the lexicon and and it wasn't just about features it was an inconsistency to there were a lot of some accounting things that Dr. and return when when when calling it there were inconsistent in itself and we have to work around that don't exactly remember what there was and that we had a lot to do all these magical was actually talking about ourselves and much more and we include all that's and accounting and accounting like the IP table stuff and so on completely into our own product and that was the most code of the you Alexi virtualization much In the 1st group monitoring instrumentation and it did you have a history of gender source so that the guy and you no genuine and angular and suicide doesn't support progressive enhancement or what is it good you know and you go to adapt the rather that so sort of progressive enhancement to use it was that of so so when you notified and you don't expect any jobs to to to to be running and the size works for rendering note the because what type of questions that might have a question that is to is uniform the contain container formats compatible with the book a while because mean if you and make so called the fork of local and if I want to use know and also think infrastructure and that didn't want to you vendor lock-in they can I move to the doctor and problem or something to say it question is if if we have a compatible and they are directories yes no we don't you could manually copy the things around and I think it would work and write some configuration files with topic which contain about which you can use it directly and of course do you do any kind of network isolation between containers of the same customer and do you that we have a network isolation all containers have a private eye tests and we only for a lot of conjugate ports for each container you can continue cell so it's like wet firewall management thing of that we don't support private networks with between continents we're here to learn and we already implemented a few things we know from people talking about downstairs but and 1 and 1 of these things is proud that and we had a reasonable use case we we have a reasonable use case for it now because of aim isn't to orchestrates applications together but to to have 1 container that contains anything you need like posters and general and whatever and but we have now a good use case where private networking is needed and I think it'll come in a short time the patent regulations literally cooperatives and that do you have any plans for an IP I want you guys to my script that these not yet very beautiful you can in fact what the side does with the this with the browser and use them but it's still session cookie and and stuff like that you can of course use their but we are on making that a lot more beautiful and document especially yes all of the data to reach this point we started in August last year the OK any final questions we have got time and the final question otherwise against the other by and all of a the
Scheduling (computing)
Computer animation
Mass
Mass
Mereology
Architecture
Broadcast programming
Computer animation
Debugger
Computer
Point cloud
Mereology
Mereology
Computer
Computer architecture
Twitter
Service (economics)
Service (economics)
Computer animation
Root
Open source
Internetworking
Server (computing)
Multiplication sign
Open source
Point cloud
Virtualization
Open set
Revision control
Summation
Computer animation
MIDI
Content (media)
Amsterdam Ordnance Datum
Template (C++)
Demon
Mobile app
Structural load
Debugger
Home page
Mereology
Uniform resource locator
Computer animation
Lipschitz continuity
Website
Message passing
Data management
Asynchronous Transfer Mode
Computer architecture
Demon
Server (computing)
Demon
Service (economics)
Server (computing)
Demo (music)
Client (computing)
Client (computing)
System call
Computer animation
Lecture/Conference
Gastropod shell
Scripting language
Video game console
Metropolitan area network
12 (number)
Raw image format
Cone penetration test
Set (mathematics)
Interior (topology)
Virtual machine
Density of states
Average
System call
Emulation
Variance
Template (C++)
Computer animation
Lecture/Conference
Commodore VIC-20
Row (database)
Scripting language
Maize
Normal (geometry)
Units of measurement
Computer-assisted translation
Hidden surface determination
Medical imaging
Server (computing)
System call
Computer animation
Virtual machine
Website
Point cloud
Virtualization
Website
Mereology
Data management
Probability density function
System call
Computer animation
Website
Point cloud
Image registration
Website
Point (geometry)
Installation art
Code
Building
Cycle (graph theory)
Cellular automaton
Magneto-optical drive
Coma Berenices
Instance (computer science)
Limit (category theory)
Mereology
Existence
Trigonometric functions
Software maintenance
Summation
Computer animation
Read-only memory
Lecture/Conference
Data management
Default (computer science)
Inheritance (object-oriented programming)
Installation art
Cycle (graph theory)
Magneto-optical drive
Coma Berenices
Instance (computer science)
Limit (category theory)
Trigonometric functions
Software maintenance
Formal language
Summation
Computer animation
Read-only memory
Lecture/Conference
Computer cluster
Resultant
Default (computer science)
Web 2.0
Computer animation
Network topology
Computer file
Hierarchy
Computer network
Vertex (graph theory)
Instance (computer science)
Line (geometry)
Family
Resultant
Template (C++)
Kernel (computing)
Computer animation
Function (mathematics)
Computer network
Counting
Point cloud
output
Central processing unit
Local Group
Table (information)
Read-only memory
Computer file
Physical law
Content (media)
Directory service
Counting
Entire function
Airfoil
Number
Cache (computing)
Computer animation
Read-only memory
Data storage device
File system
Statistics
Data storage device
Point (geometry)
Computer animation
Network topology
Computer file
Sampling (statistics)
Volume (thermodynamics)
Point (geometry)
Video game
Computer animation
Multiplication sign
Computer file
MiniDisc
Volume (thermodynamics)
Selectivity (electronic)
Volume
Subtraction
10 (number)
Computer file
Block (periodic table)
Computer file
Physicalism
Virtualization
Volume (thermodynamics)
Medical imaging
Computer animation
Bit rate
Personal digital assistant
Logic
File system
MiniDisc
Diagram
Right angle
Software testing
Data management
Partition (number theory)
Units of measurement
Form (programming)
Area
Point (geometry)
Vapor barrier
Block (periodic table)
State of matter
Point cloud
Set (mathematics)
Writing
Category of being
Hypermedia
Computer animation
Bit rate
Hypermedia
File system
Codec
Block (periodic table)
Vapor barrier
Writing
Point (geometry)
Partition (number theory)
Vapor barrier
Computer animation
Network topology
Mathematical singularity
Software testing
RAID
Stability theory
Demon
Read-only memory
Block (periodic table)
Home page
Line (geometry)
Vector potential
Uniform resource locator
Kernel (computing)
Resource allocation
Computer animation
Read-only memory
Personal digital assistant
Order (biology)
Analytic continuation
Read-only memory
Order (biology)
Kernel (computing)
Resource allocation
Computer animation
Network topology
Read-only memory
State of matter
Infinite conjugacy class property
Home page
Bit
Block (periodic table)
Read-only memory
Message passing
Serial port
Event horizon
Computer animation
Computer file
Point cloud
Event horizon
Leak
Spacetime
Computer animation
Multiplication sign
String (computer science)
Interior (topology)
MiniDisc
Limit (category theory)
Limit (category theory)
Mereology
Vector potential
Number
Computer animation
Content (media)
Bit
Twitter
Metropolitan area network
Formal language
Email
Symbol table
Computer animation
Torque
HTTP cookie
Default (computer science)
Pairwise comparison
Standard deviation
Process (computing)
Scripting language
Software developer
Library catalog
Template (C++)
Formal language
Formal language
Template (C++)
Symbol table
Computer animation
Information
Default (computer science)
Scripting language
View (database)
State of matter
View (database)
Computer file
Java applet
Library catalog
Variance
Chaining
Medical imaging
Process (computing)
Computer animation
Information
Domain name
Touchscreen
Service (economics)
Inclusion map
Standard deviation
Service (economics)
Computer animation
Authentication
Dependent and independent variables
Login
Exception handling
Formal language
Default (computer science)
Web page
Trail
Computer animation
Oval
Personal digital assistant
Function (mathematics)
Authentication
Factory (trading post)
Heat transfer
Dependent and independent variables
Image registration
Window
Web page
Trail
Trail
Computer animation
Open source
Electronic mailing list
Heat transfer
Point cloud
Code
View (database)
Decision theory
Multiplication sign
Conjugacy class
Social class
Process (computing)
Product (category theory)
Trail
Regulator gene
File format
Software developer
Virtualization
Element (mathematics)
Arithmetic mean
Process (computing)
Module (mathematics)
Configuration space
Quicksort
Data type
Arithmetic progression
Data management
Point (geometry)
Computer programming
Read-only memory
Server (computing)
Open source
Computer file
Firewall (computing)
Division (mathematics)
2 (number)
Revision control
Goodness of fit
Software testing
Scripting language
Statement (computer science)
Analytic continuation
Addition
Server (computing)
Gender
Cellular automaton
Consistency
Planning
Directory service
Binary file
Cartesian coordinate system
Local Group
Table (information)
Computer animation
Personal digital assistant
Computer network
Key (cryptography)
HTTP cookie
Loading...
Feedback

Timings

  542 ms - page object

Version

AV-Portal 3.8.0 (dec2fe8b0ce2e718d55d6f23ab68f0b2424a1f3f)