Add to Watchlist

Mobile Games to the Cloud With Python

27 views

Citation of segment
Embed Code
Purchasing a DVD Cite video

Automated Media Analysis

Beta
Recognized Entities
Speech transcript
hello everyone and that could end users behind the Huber joining a bit later so from their from 9 its operation company and they
will be talking about mobile games so the clouded by tomorrow how we as a company made transaction furthermore the more mobile from a mobile base gaming to 1 that is convex so I will divide this talk into 3 parts the 1st part remind me then this will take over and then finish so in size so how we this to talk about be divided so we have therefore the into and the requirement that by the vector you will see that later but I want to what the objectives of 1st objective is to show you how we connect the basic knowledge is used by to how we created a scalable architecture as a result of it and how we using Python in our company a a number of places but not the 2nd objective which I find more important is to show you the lessons we learned the challenges we face the difficulties we had and what the problems were when creating such a system that has a lot of
customers that has a lot of the players which was which are on smartphones and which problems arise which mark dating and stuff like that so who are know just talk intruders don't short part about the company we are not of the creation and development company and we
started in 2 thousand and 9 and up till until 2011 growing developing applications and
then we started developing because I in that time the games were a more profitable and more interesting part of the industry so uh after with which the games we decided OK our games are on this smart but we need some kind of back and so on so on the services which are going to
connect the where's the fun games together like a social part of the so we started developing beckoned server in Python and after that we all resources the growth we had all the games we develop connected this beckoned but after some time that they can be back and became too slow and too it wasn't sufficient enough for our new requirements on the in our new games so you can see we are using our model and we have a lot of instances and we have a lot of requests daily and that's rising every day and every year so how do you fight in the past like I said need some sort of social features so we started with our 1st iteration of several with which was pretty easy we used simple server in python and that
that last for about a month and then we realized that that won't work we switch to a more robust and more proof solution which was a partially with gender and that was case for requirements Baghdad our requirements only to connect the players in a social way so they can they can view each other's mind games they can use send gifts to each other and stuff like that after that we added some more functionalities like objects in the games with vacation settings and stuff like that and we ended up secondary several which was for our in-house analytics systems and that was working pretty well but we only had 1 big issue we had only vertical scalability so every 12 months are server will become too too small and would would be able to handle the stress and the number of
requests so we had to but to bump up the the server and that lasted for 2 years every 12 months of bigger server and we wanted to have horizontal scalability but couldn't because we had a problem with same games which were on which were which were saved from the bees and this would not allow us to have horizontal scalability so a change was needed because of that so why did we make the change so we have a new game this new game in your
requirements and these new requirements world OK let's have a better let's have players who play against each other and when you have a partially that's not possible because the party is a brief worker based web server which doesn't allow you a lot of consistent persistent connections so we had to change that's another thing we had my sequel my sequel wasn't good enough for stuff like that you couldn't we had problems with my sequel when we develop the analytics system because we had like 50 60 million million the roles of data in 1 table and we had to join that didn't work so we said OK we gonna change and we gonna make up solution that is more scalable that would allow us not only to have the social part of the game but the entire multiple
and 1 other thing we decided OK we're gonna change we gonna put the game logic of the game on the back and no longer will the game be played only on the smartphone is gonna be played on the on the back and from now on so a player would send a request to the server every time something happens in the game so that we can have the a same on the server that we can have players that played on multiple devices and when you have saved games on devices like we had in the beginning you have problems with cheating because it is this was a player can reach its own his own stadium it and she he can just change the number of like he has to 4 billion you can change the price of some stuff in the game to minus 2 billion and when he buys this stuff he gains 2 billion points so that that was an issue so we decided we have to change that we have to we have to stop the players from cheating so what was the goal of the goal was to have a system that was scale horizontally every part of the system that was primarily primary part system with scale horizontal we had wanted to have an automatic deployment because of fuel now we have to uphold the source code is each sent to the server and that was not feasible in we had to properly had to restart the server the Apaches time we decided to change and we wanted to containerized each part of our back and so like we have layers of back and we wanted to have them transferable easily transferable between the instances between service since I like it said the game the game logic had to be transferred to the back and that meant that each player has to have a persistent connection to the server and eparchy like if that wasn't good enough for that so we had to find a solution how to fill have like 10 20 30 thousand of players which place simultaneously in each of these players have our Pryce's connection to the server so and other things up till now until now game with full new data from the server it would send a request it would receive a response from the server we want to change you wanted to have the server portion of data to the to the games the client each time a new data would appear so the we decided to use sort of sort of our events based architecture we had a number of requests that would be sent in a branch from the client if those requests will not go true there would be a and again in like 3 4 seconds at that that the to have to have to so the data would not be lost in transactions into in the transfer and stuff like that and the cheating it makes cheating very difficult because we tried to to each request that will up alive so that would arrive on the back and would be check to see if this request is of it is possible what happens if a player wanted to buy something in the game we will check if he did this has enough points has enough to any the other enough gems sort with or any other in and other this resource in so right which was quite we already had quite like you
generally use a lot of things can and that we can i for us and we wanted to continue that way and like I said we needed a server that will allow us to have a large number of persistent connections and tornado tomato fits perfectly here because it's made for that is made for persistent connections from the clients and after a couple of thousand connections to want major instance we said OK we can start another instance we can have a a proxy in the from in front of that that little balance between we wanted automatic deployment fabric is also great for that we had fabric connects to our Muslim using 2 instances check for it each type of
instances which states and we used both before that you could see both on the slide to and we had workers workers were part of a system that would that would process the requests that arrive from the smart from the game since we have a working gender server we wanted to have some sort of backward compatibility so we can reuse the database that we can use the cold and we can use the stuff we already had that was required for the new game and we have a lot of database we have 3 databases and they all had great frameworks invited so Python what the solution for that is OK we can go with Python from from this from this spot on and continue the way it was used until the so I'm going to leave it to me something that I want to the architecture and the parts of the
system so also the architecture better and the diagram about from now on the diagram of the architecture is divided into several layers approach to the rules that
were requested web servers the web servers voltage accused which told you the workers and the total today's databases and send a reply back continues whose onto a scalable and everybody needs to be
scalable and easily you can use like it is Boca eyes so we of that makes deployment really isn't supplied vitamins great here because I have only to we developed the cold so that implements the system was the game logic but also but it was a great binding although services together with its libraries so everything is going up virtual private clouds so I can only
do brought to the engine structures available to the
outside you can see how that travels through through the system and the lines represent the inbound traffic and yellow lines represent all about traffic inbound genetic counseling year http requests and outbound traffic goes into the units as you can see we will use the the Muslim RDFS and this is the for the when web server parts In is difficult context consists of 2 2 parts the it later which receives requests and the WebSocket merit which pushes the responsibility of the game the engine makes use IP caching so the persistent connections are possible and WebSockets WebSockets can survive the when request Council goes the WebSocket to the web server and used to our in-house built you and the responses the responses come through radius and use releases acuities simplest and their alone WebSockets when the and a list of July requests our response formats that give us some problems but we'll talk about later now the workers a singleton Python programs book arise in the connected to we had to use our current document because of some issues which react with talk about that later in the and there is still a requests from an RQ and message from client has unique message type and when we develop new again so we just have to process different events that are specific to that game and the rest of the system I Q is written in going
which we can write the cube because of an issue we will react when because of its architecture doesn't support transactions so when multiple multiple points right to the same data points they can create what they call siblings which of pieces of data with different reflux and have to manage that better yourself we saw that issue by having a q which routes requests for single users of connections always the same work we use the caching for that and the so that we can make sure that every piece of data that is written to is written by the same worker and there will never of rights multiple times same data points the the cue detects when workers go online and offline everything in architecture is made that everything is detectable and messages look you up until the world has gone up or it will and starting from number of of so technology use and makes
for the for the web server 0
and q is used in several places so that we're going to turn off and several components and look the messages will go through same for the databases we use the credits and was that the everything we look for all the components we just have to to make was 1 master and the also receive who will build docker containers of the fabric and with the instances which have to be the start of the new images available it will pull the image receptor blocker and with an input running and so as to optimize the full original spilled into so powerful so as you can see we used Python to bind various acknowledges together that works great and as you can see we have a web server in python had workers and by Python and we have them connecting various databases clients together but I'm going to talk about to the lessons learned because when developing such a system you would face difficulties and we had a number of difficulties some more specific to the smartphone part of the industry so 1 of the 1st things mobile devices have mobile connections mobile connections can be slow when a person walks to the to the to the streets sometimes it disconnects from 1 power connects to another power it connects to the feed it disconnects from so you have connections which drop and new connections which start and we had had a problem because the TCP doesn't help you when you try to discover connection drops so we had to fix that by monitoring our connections frequently we had to being from the client to the server we had to being from the service to the client white because each time a new connection which forest floor and it was still some of the data which arrives to and that was the problem so we used but publish and subscribe red to figure that issue we had to do inter tomato communicate between instances because 2 we had multiple tornadoes and when a connection with drought no other natives of would not that it it has dropped so that was a big issue it's still an issue and I think this is a problem with most of most connections because they drop frequently are all our old system also had this type of problems but Apache would fix for us you had like when a player leaves the game it
would leave a hand in connection with food drop only in 3 or 4 minutes but that wasn't the issue that now it is an issue we had to fix it with multiple things multiple a multiple places and that was the biggest issue on other issues with the use of self react and race conditions INRIA so really does not allow you to have it allows you to have multiple workers right to the same data but when you do you have a condition which you have to fix readjusts tells you OK you have a condition have 2 siblings now you have to decide which data is the correct 1 you have to merge the data yourself and that was initially to us because if we if 2 workers right data to react then the workers would have to know how to fix the issues that arise and since we have a lot of requests that come from from the games each request is from his own request type we had to know how to fix each request each request type and that was a big problem so we decided OK we're not just skip the problem we going pretend the problem isn't there we gonna fix it like said you don't have 1 worker you see all the data from 1 client and this data would arrive in the same order and 1 workers with only right for she for the his clients and that would allow the client only to have 1 record for him and that fixed the issue because there were no multiple workers right the same data so another thing sequel data when we use equal blocks are single threaded nature of our work and since workers have to process a lot of requests we had to make sure that it be run as fast as possible so we decided OK we're going dropped below would push some data to additional threats which would write sequel only when there are no other data process we decided that these data is low priority some of the sequel that is low priority and it can be written in 2 3 or 4 seconds after the request of processed so the main matrix will not block the main will be able to process as much as it was possible and that in yeah I would increase the throughput of our system another thing for NATO and
rents they don't work together in quite well because the neighborhood of not allow you to block anything it has to go out of the stuff is it is synchronous if you try to block in red is you have to block on the server side and if you block like on a block all what ball or not on subscribe it would bother to neighbors of complete so there were a couple of solutions but both solutions we tried some issues so we had figures of features we had issues when you try to disconnect from and stuff like that so we had to implement some new stuff on that part and that was problem and other things are really we have a real of 5 databases and we we had 1 problem when 1 area instance pressure and we didn't know about that because we were in the middle of development and we shall we had this 1 server crashed for about 1 month and everything was working fine because the area shield itself magically we didn't know anything happened that to what they we try to connect to this incident that was down and we we realize that it wasn't responding to peaks anything but the data was secure data wasn't corrupted everything was working as it as it was supposed to so what's in the future like like we said the dog right every container possible and talker is a great new technology and I think you have a great talk tomorrow by our colleagues which will which will show you how to use the local in a particular development environment
and I would suggest you listen to that because it allows you to some great stuff especially on the production like this you can transfer probably images of of your property of cold between service and Dr. now has a native Python support so you can write Boca broke Boca containers from your Python code which you have for automatic deployment and stuff like that another thing we're working on these automated automatic testing facility which uses a lot of doctors and it allows you to use all of us to run up to on a new iteration of our web service up our entire back and to see if each request that arise with always yield the same response and since we have 2 connections currently we have 1 connection which is http which has receiving requests and we have a let's look at that is only for the responses to the game we want to unify these 2 1 single connection because because it would be much easier than it is now this also brings us some issues but we're trying to fix it with 1 connection and 1 interesting thing when we developed the 1st game using this back and it's the required 5 months to do it because we had to write the entire back and from scratch but the 2nd game was finished in Mountain have because we only had to transfer the game logic from the game to to that could be only already had the entire working back and the web service that you lose the workers and we only had to transfer how to process the requests that arrive to the server and that that would allow was found to have our new games which would which would be developed also need a lot less time than the 1st thing so that's that's it you can ask general some questions if you want to think
action well we send it to an adjacent form with yeah the the the caller was asking how do we send the data from the from the client to the server so how it this how you performance so we use Jason to form of the data we try to we wait for for like 3 seconds to see all the requests that are going to be sent to the server and then we send it in a batch if this better goes through in yeast marked as completed and it even send again if it isn't received up the flow if the message is received that is processed descend again in next batch but the next page always has the same order of messages like they were produced in the game so it basically we have a adjacent data which has requests which are requested that were generated in the last 3 4 seconds that word processors this is 1 type and the other type of messages that are critical high-priority they are sent instantaneously in the the 1 message those messages like our stuff like a player tries to take another place where a player tries to requests the current leader board and stuff like that so data that is needed as soon as possible well we we didn't have it for the 1st game but we're doing it right now you have always so the column was that he was method of testing and staging in our system so we wouldn't have it at the beginning but now we're doing this automated testing which would allow us that as soon as we do it it would we would spin off a new back in and it would have have unit test for each for each request that that that is in our system and it would try to you to start some use cases in the game and it it then would which tried to run use cases and see if the responses in the database in the game are as they are supposed to be higher authority also the in the US so using you run for office the uses a lot of time well the question is why did we use we use both 0 and q and red this and why did decide to use the register as a public Publish-Subscribe Service and that's you and your so basically when we started using 0 and you we try to use it in place of the custom in-house q we developed and we were we realized that it didn't support what we need and we already had red is doing this stuff and since registers very powerful it allows you to do blocking books published by ASCII values storage everything you need we decided that as we already using red for this kind of stuff we can use it as a as a publish-subscribe so best probably the main reason 0 and and as much as it isn't so much used in our system as registers so there's a there's a reason I have my hand all of them you all and so on so basically the question is why did we use a neighborhood and if we try to use other technologies such as that so could you and stuff and so when we started developing we decided we knew server that would support a lot larger and larger number of persistent connections and the choice we did some research with this look for the imitation and some public that and we decided that enable in that point was the best solution because his head whose great support and it has connections with the which tornado of the selected with the red these connections with 0 and Q and that was what we needed at that time so basically that was the reason we do it but we are currently looking into finding in alternative because we're trying aggressively trying to unify the connections only have 1 connection and that would allow us to have to try new technologies and see if something else would fit into the shoes of the tools that enable fits right now so all of yes here on you know that's a great question and the repeated so the the question is how do we point service because when you deploy web service and have if you had a persistent action this persists connection would drop and what will the client would then send a request to a server that doesn't exist basically so we have that the client is uh for bulletproof so say state to connection drops if a connection drops the client would know immediately because when the but then genetics and from a drop it drove the entire TCP connections and the client would then reconnect immediately to when you but server which is deployed automatically in that's like 2nd order to delay but those things we don't have to have recourse responses that the immediate we can better we can send messages in batch and every 4 5 seconds that isn't of a nation issues so we decided we can stop the local containers and stop them again in 1 2nd and the client would not know that anything happened it would just be coming together this is because of all of the OK the question is why did use the sockets and only for the for the returning vitamin mean yeah because we wanted to have a persistent connections and 1 of the solutions was to have a low level TCP connections but we wanted to have a connection that is more of a higher level sort of like http because our requests that arrive to the server Our also HTTP connections so we decided to use that soccer's because they're up and operate to our current architecture that we already use and they're not going to do the connection of already use for receiving requests and doing we needed something that would that would have stream-like connections and the socket fit the bill perfectly here so with this loss
Mobile Web
Database transaction
Bit
Mereology
Number
Computer animation
Vector space
Convex set
Operator (mathematics)
Object (grammar)
Game theory
Game theory
Resultant
Computer architecture
Physical system
Content (media)
Computer animation
Software developer
Smartphone
Mereology
Game theory
Server (computing)
Server (computing)
Multiplication sign
Scientific modelling
Heat transfer
Instance (computer science)
Mereology
Flow separation
Web service
Computer animation
Game theory
Quicksort
Game theory
Server (computing)
Server (computing)
Gender
Analytic set
Set (mathematics)
Binary file
Scalability
Functional (mathematics)
Scalability
Architecture
Mathematics
Personal digital assistant
Stress (mechanics)
System on a chip
Game theory
Object (grammar)
Physical system
Game theory
Physical system
World Wide Web Consortium
Point (geometry)
Server (computing)
Sequel
Multiplication sign
Source code
Branch (computer science)
Instance (computer science)
Heat transfer
Client (computing)
Mereology
Event horizon
Scalability
Number
Web 2.0
Architecture
Web service
Mathematics
Software
Logic
Message passing
Game theory
Multiplication
Sanitary sewer
Physical system
Computer architecture
Server (computing)
Consistency
Analytic set
Mereology
Instance (computer science)
Group action
Connected space
Table (information)
Computer animation
Logic
Mathematics
Dependent and independent variables
Smartphone
Game theory
Quicksort
Slide rule
Server (computing)
State of matter
Mereology
Number
Telecommunication
Database
Logic
Software framework
Message passing
Proxy server
Game theory
Physical system
Computer architecture
World Wide Web Consortium
Chatterbot
Axiom of choice
Torus
Server (computing)
Chemical equation
Gender
Instance (computer science)
Connected space
Computer animation
Database
Software framework
Queue (abstract data type)
Game theory
Quicksort
Server (computing)
Proxy server
Server (computing)
Total S.A.
Scalability
Rule of inference
Web 2.0
Architecture
Computer animation
Database
Database
Diagram
Queue (abstract data type)
Computer architecture
Physical system
World Wide Web Consortium
Server (computing)
Point cloud
Scalability
Architecture
Web service
Computer animation
Database
Logic
Queue (abstract data type)
Game theory
Data structure
Physical system
Library (computing)
World Wide Web Consortium
Point (geometry)
Teilnehmerrechensystem
Computer programming
Database transaction
Server (computing)
Context awareness
Multiplication sign
Client (computing)
Mereology
Event horizon
Web 2.0
Architecture
Subtraction
Units of measurement
Physical system
Computer architecture
Multiplication
File format
Uniqueness quantification
Electronic mailing list
Line (geometry)
Connected space
Software development kit
Message passing
Radius
Computer animation
Uniform resource name
Dependent and independent variables
Natural language
Right angle
Game theory
Data type
Vacuum
Wide area network
Server (computing)
Building
Proxy server
Multiplication sign
Connectivity (graph theory)
Online help
Client (computing)
Drop (liquid)
Mereology
Power (physics)
Number
Web 2.0
Architecture
Medical imaging
Connected space
Latent heat
Web service
Telecommunication
Database
Forest
Smart card
Data storage device
Message passing
Physical system
World Wide Web Consortium
Mobile Web
Server (computing)
Keyboard shortcut
Instance (computer science)
Thread (computing)
Connected space
Message passing
Computer animation
Database
Web service
Crash (computing)
output
Smartphone
Queue (abstract data type)
Key (cryptography)
Game theory
Data type
Server (computing)
Sequel
Drop (liquid)
Client (computing)
Complete metric space
Mereology
2 (number)
Matrix (mathematics)
Telecommunication
Natural number
Database
Lattice (order)
Smart card
Data storage device
Uniform boundedness principle
Message passing
Condition number
Physical system
Area
Electronic data processing
Hidden surface determination
Block (periodic table)
Software developer
Neighbourhood (graph theory)
Instance (computer science)
Incidence algebra
Thread (computing)
Connected space
Integrated development environment
Crash (computing)
Order (biology)
Condition number
Right angle
Game theory
Figurate number
Pressure
Data type
Sinc function
Row (database)
Axiom of choice
Group action
Code
State of matter
Multiplication sign
Insertion loss
Client (computing)
Disk read-and-write head
Faculty (division)
Word processor
Web service
Network socket
Damping
Office suite
Physical system
Product (category theory)
Unit testing
Entire function
Connected space
Category of being
Message passing
Arithmetic mean
Data storage device
Order (biology)
Software testing
Quicksort
Whiteboard
Data type
Sinc function
Point (geometry)
Ocean current
Web page
Dataflow
Server (computing)
Socket-Schnittstelle
Batch processing
Drop (liquid)
2 (number)
Number
Database
Authorization
Energy level
Software testing
Game theory
Computer architecture
World Wide Web Consortium
Server (computing)
Computer animation
Logic
Personal digital assistant
Dependent and independent variables
Iteration
Game theory

Metadata

Formal Metadata

Title Mobile Games to the Cloud With Python
Title of Series EuroPython 2014
Part Number 45
Number of Parts 120
Author Stipetic, Mislav
Ronić, Darko
License CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
DOI 10.5446/20009
Publisher EuroPython
Release Date 2014
Language English
Production Place Berlin

Content Metadata

Subject Area Information technology
Abstract Mislav Stipetic/Darko Ronić - Mobile Games to the Cloud With Python When a mobile game development company decides to switch to a more cloud based development it is faced with obstacles different from those it’s used to on mobile devices. This talk explains how Python provided us with most of the infrastructure for this task and how a Python game backend was built as a result. ----- #### The Talk This talk has two goals. Showing the audience the lessons we learned during a project which moved a simple mobile game to a server backend is our first intention. In addition to that we want to describe how such a system works in a real life example, to show which problems and which requirements arise in its creation. When the audience leaves the talk they will know how a real-life mobile game uses Python for powering the backend servers. #### The Problem Most of the game development for mobile devices is focused on running the game on the device. The game designers and game developers play a primary role in creating the product. The server backend plays a supporting role providing a multiplayer or social experience to the users. Indeed, at Nanobit Ltd., things were also done that way. We had a small Python infrastructure built around Django which provided a small portion of multiplayer experience for the players. The majority of development was still focused on playing the game on the device. That way of thinking was put to test when we decided to center our future games around the multiplayer experience. Due to the fact that our infrastructure at the time was not enough for what we had in mind, we had to start from scratch. The decision was made to use Python as the center of our new infrastructure. In order to achieve it, a server backend was needed that would allow the game to be played “in the cloud” with the device only being a terminal to the player. Most of the game logic would have to be processed in the cloud which meant that each player required a constant connection to the backend and with over 100.000 players in our previous games that presented a challenge. How to build an infrastructure which can support that? Since every user action had to be sent to the backend how to process thousands of them quick enough? Those problems were big and were just the start. #### The Solution The design of the backend lasted for a couple of months and produced a scalable infrastructure based on “workers” developed in Python, “web servers” that use Tornado and a custom message queue which connected the two. The storage part is a combination of Riak and Redis. Since the backend is scalable new workers and new web servers had to be deployed easily so an orchestration module was build using Fabric. The scalability and launching of new workers and web servers was achieved using Docker for creation and deployment of containers. Each container presents one module of the system (worker, web server, queue). The end result can now support all of our future games and only requires the game logic of each game to be added to the workers. #### The Technologies * Python for coding the game logic, web servers. More than 90% of the system was written in Python. * Fabric * SQLAlchemy * Riak * Redis * ZeroMQ * nginx * Docker * Websockets * AWS #### The Lessons Learned * How to tune the backend to handle the increasing number of active players. * How to tackle the problem of frequent connection dropping and reachability issues of poor mobile device Internet connection in Tornado with a little help of Redis. * How to prevent users from trying to outsmart the system by denying illegal moves. * How to enable game profile syncing and live updating. * Improving the performance of workers by prioritizing data being stored to databases (Riak, SQL). * New issues and lessons show up all the time so there will definitely be more of them by the time of the conference. #### Basic Outline 1. Intro (5 min) 1. Who are we? 2. How was Python used in our previous games 3. Why we decided to change it all 2. Requirements (6 min) 1. What was the goal of creating the game backend 2. Why was Python our first choice 3. Python backend (14 min) 1. The architecture of the backend 2. Which technologies did we use and how were they connected together 3. How the backend handles the game logic 4. Lessons learned 4. Questions & Answers (5 min)
Keywords EuroPython Conference
EP 2014
EuroPython 2014

Recommendations

Loading...
Feedback
AV-Portal 3.5.0 (cb7a58240982536f976b3fae0db2d7d34ae7e46b)

Timings

  600 ms - page object